overriding logback.xml apexApps.sh: running application 'onappf' with command 'java -Dlogback.configurationFile=/opt/app/policy/apex-pdp/etc/logback.xml -cp /opt/app/policy/apex-pdp/etc:/opt/app/policy/apex-pdp/etc/hazelcast:/opt/app/policy/apex-pdp/etc/infinispan:/opt/app/policy/apex-pdp/lib/* -Djavax.net.ssl.keyStore=/opt/app/policy/apex-pdp/etc/ssl/policy-keystore -Djavax.net.ssl.keyStorePassword=Pol1cy_0nap -Djavax.net.ssl.trustStore=/opt/app/policy/apex-pdp/etc/ssl/policy-truststore -Djavax.net.ssl.trustStorePassword=Pol1cy_0nap -Dlogback.configurationFile=/opt/app/policy/apex-pdp/etc/logback.xml -Dhazelcast.config=/opt/app/policy/apex-pdp/etc/hazelcast.xml -Dhazelcast.mancenter.enabled=false org.onap.policy.apex.services.onappf.ApexStarterMain -c /home/apexuser/config/OnapPfConfig.json' [2022-10-10T11:38:20.062+00:00|INFO|ApexStarterMain|main] In ApexStarter with parameters [-c, /home/apexuser/config/OnapPfConfig.json] [2022-10-10T11:38:20.728+00:00|INFO|SingleThreadedDmaapTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=false, locked=false, uebThread=null, topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: INITTED [2022-10-10T11:38:20.835+00:00|INFO|ServiceManager|main] service manager starting [2022-10-10T11:38:20.835+00:00|INFO|ServiceManager|main] service manager starting topics [2022-10-10T11:38:20.845+00:00|INFO|SingleThreadedBusTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=false, locked=false, uebThread=null, topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: starting [2022-10-10T11:38:20.847+00:00|INFO|SingleThreadedDmaapTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=false, locked=false, uebThread=null, topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: INITTED [2022-10-10T11:38:20.851+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:38:20.856+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:38:20.848+00:00|INFO|InlineBusTopicSink|main] InlineDmaapTopicSink [userName=null, password=null, getTopicCommInfrastructure()=DMAAP, toString()=InlineBusTopicSink [partitionId=74cfd6cf-bfa3-452c-bed4-d6285736bf58, alive=false, publisher=null]]: starting [2022-10-10T11:38:20.868+00:00|INFO|InlineDmaapTopicSink|main] InlineDmaapTopicSink [userName=null, password=null, getTopicCommInfrastructure()=DMAAP, toString()=InlineBusTopicSink [partitionId=74cfd6cf-bfa3-452c-bed4-d6285736bf58, alive=false, publisher=CambriaPublisherWrapper []]]: DMAAP SINK created [2022-10-10T11:38:20.869+00:00|INFO|ServiceManager|main] service manager starting set alive [2022-10-10T11:38:20.869+00:00|INFO|ServiceManager|main] service manager starting register pdp status context object [2022-10-10T11:38:20.872+00:00|INFO|ServiceManager|main] service manager starting topic sinks [2022-10-10T11:38:20.872+00:00|INFO|ServiceManager|main] service manager starting Pdp Status publisher [2022-10-10T11:38:20.874+00:00|INFO|ServiceManager|main] service manager starting Register pdp update listener [2022-10-10T11:38:20.875+00:00|INFO|ServiceManager|main] service manager starting Register pdp state change request dispatcher [2022-10-10T11:38:20.875+00:00|INFO|ServiceManager|main] service manager starting Message Dispatcher [2022-10-10T11:38:20.875+00:00|INFO|TopicBase|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=0, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=0]]]]: registering org.onap.policy.common.endpoints.listeners.MessageTypeDispatcher@2ed3b1f5 [2022-10-10T11:38:20.876+00:00|INFO|SingleThreadedBusTopicSource|main] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: register: start not attempted [2022-10-10T11:38:20.876+00:00|INFO|ServiceManager|main] service manager starting Create REST server [2022-10-10T11:38:20.876+00:00|WARN|PdpMessageHandler|Timer-0] not registered: object:engine/apex/handler [2022-10-10T11:38:21.021+00:00|INFO|OrderedServiceImpl|Timer-0] ***** OrderedServiceImpl implementers: [] [2022-10-10T11:38:21.025+00:00|INFO|network|Timer-0] [OUT|DMAAP|POLICY-PDP-PAP] {"pdpType":"apex","state":"PASSIVE","healthy":"HEALTHY","description":"Pdp Heartbeat","statistics":{"pdpInstanceId":"apex-d89fe811-168c-4a33-8281-85b7ca1b5a33","timeStamp":"2022-10-10T11:38:20.972210Z","pdpGroupName":"defaultGroup","policyExecutedCount":0,"policyExecutedSuccessCount":0,"policyExecutedFailCount":0,"policyDeployCount":0,"policyDeploySuccessCount":0,"policyDeployFailCount":0,"policyUndeployCount":0,"policyUndeploySuccessCount":0,"policyUndeployFailCount":0},"messageName":"PDP_STATUS","requestId":"0d6c8ead-3971-43ec-bd98-f6db271305a9","timestampMs":1665401900875,"name":"apex-d89fe811-168c-4a33-8281-85b7ca1b5a33","pdpGroup":"defaultGroup"} [2022-10-10T11:38:21.071+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 37 ms [2022-10-10T11:38:21.075+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:21.078+00:00|INFO|log|main] Logging initialized @1805ms to org.eclipse.jetty.util.log.Slf4jLog [2022-10-10T11:38:21.081+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:21.081+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:21.089+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:21.090+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:38:21.093+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:21.093+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:21.249+00:00|INFO|ServiceManager|main] service manager starting Rest Server [2022-10-10T11:38:21.249+00:00|INFO|ServiceManager|main] service manager starting [2022-10-10T11:38:21.249+00:00|INFO|ServiceManager|main] service manager starting REST RestServerParameters [2022-10-10T11:38:21.249+00:00|INFO|JettyServletServer|main] JettyJerseyServer [Jerseyservlets={/metrics=io.prometheus.client.exporter.MetricsServlet-3c1e3314==io.prometheus.client.exporter.MetricsServlet@de1cbbac{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-2c282004==org.glassfish.jersey.servlet.ServletContainer@f07d2247{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-66d23e4a==io.swagger.jersey.config.JerseyJaxrsConfig@b6fbab00{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}}, swaggerId=swagger-6969, toString()=JettyServletServer(name=RestServerParameters, host=0.0.0.0, port=6969, user=healthcheck, password=zb!XztG34, contextPath=/, jettyServer=Server@37ff4054{STOPPED}[9.4.48.v20220622], context=o.e.j.s.ServletContextHandler@894858{/,null,STOPPED}, connector=RestServerParameters@2db2cd5{HTTP/1.1, (http/1.1)}{0.0.0.0:6969}, jettyThread=null, servlets={/metrics=io.prometheus.client.exporter.MetricsServlet-3c1e3314==io.prometheus.client.exporter.MetricsServlet@de1cbbac{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-2c282004==org.glassfish.jersey.servlet.ServletContainer@f07d2247{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-66d23e4a==io.swagger.jersey.config.JerseyJaxrsConfig@b6fbab00{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}})]: STARTING [2022-10-10T11:38:21.276+00:00|INFO|ServiceManager|main] service manager started [2022-10-10T11:38:21.276+00:00|INFO|ServiceManager|main] service manager started [2022-10-10T11:38:21.276+00:00|INFO|ApexStarterMain|main] Started policy-apex-pdp service successfully. [2022-10-10T11:38:21.276+00:00|INFO|JettyServletServer|RestServerParameters-6969] JettyJerseyServer [Jerseyservlets={/metrics=io.prometheus.client.exporter.MetricsServlet-3c1e3314==io.prometheus.client.exporter.MetricsServlet@de1cbbac{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-2c282004==org.glassfish.jersey.servlet.ServletContainer@f07d2247{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-66d23e4a==io.swagger.jersey.config.JerseyJaxrsConfig@b6fbab00{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}}, swaggerId=swagger-6969, toString()=JettyServletServer(name=RestServerParameters, host=0.0.0.0, port=6969, user=healthcheck, password=zb!XztG34, contextPath=/, jettyServer=Server@37ff4054{STOPPED}[9.4.48.v20220622], context=o.e.j.s.ServletContextHandler@894858{/,null,STOPPED}, connector=RestServerParameters@2db2cd5{HTTP/1.1, (http/1.1)}{0.0.0.0:6969}, jettyThread=Thread[RestServerParameters-6969,5,main], servlets={/metrics=io.prometheus.client.exporter.MetricsServlet-3c1e3314==io.prometheus.client.exporter.MetricsServlet@de1cbbac{jsp=null,order=-1,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /*=org.glassfish.jersey.servlet.ServletContainer-2c282004==org.glassfish.jersey.servlet.ServletContainer@f07d2247{jsp=null,order=0,inst=false,async=true,src=EMBEDDED:null,STOPPED}, /=io.swagger.jersey.config.JerseyJaxrsConfig-66d23e4a==io.swagger.jersey.config.JerseyJaxrsConfig@b6fbab00{jsp=null,order=2,inst=false,async=true,src=EMBEDDED:null,STOPPED}})]: STARTING [2022-10-10T11:38:21.282+00:00|INFO|Server|RestServerParameters-6969] jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.15+10-alpine-r0 [2022-10-10T11:38:21.317+00:00|INFO|session|RestServerParameters-6969] DefaultSessionIdManager workerName=node0 [2022-10-10T11:38:21.317+00:00|INFO|session|RestServerParameters-6969] No SessionScavenger set, using defaults [2022-10-10T11:38:21.319+00:00|INFO|session|RestServerParameters-6969] node0 Scavenging every 660000ms WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.sun.xml.bind.v2.runtime.reflect.opt.Injector (file:/opt/app/policy/apex-pdp/lib/jaxb-impl-2.3.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int) WARNING: Please consider reporting this to the maintainers of com.sun.xml.bind.v2.runtime.reflect.opt.Injector WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release [2022-10-10T11:38:22.119+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 1085 ms [2022-10-10T11:38:22.119+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:22.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:22.136+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:22.137+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:22.137+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:22.195+00:00|INFO|GsonMessageBodyHandler|RestServerParameters-6969] Using GSON for REST calls [2022-10-10T11:38:22.202+00:00|INFO|YamlMessageBodyHandler|RestServerParameters-6969] Accepting YAML for REST calls [2022-10-10T11:38:22.326+00:00|INFO|ContextHandler|RestServerParameters-6969] Started o.e.j.s.ServletContextHandler@894858{/,null,AVAILABLE} [2022-10-10T11:38:22.346+00:00|INFO|AbstractConnector|RestServerParameters-6969] Started RestServerParameters@2db2cd5{HTTP/1.1, (http/1.1)}{0.0.0.0:6969} [2022-10-10T11:38:22.347+00:00|INFO|Server|RestServerParameters-6969] Started @3073ms [2022-10-10T11:38:23.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 2135 ms [2022-10-10T11:38:23.170+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:23.170+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:23.175+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:23.176+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:23.176+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:24.218+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 3184 ms [2022-10-10T11:38:24.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:24.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:24.222+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:24.223+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:24.223+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:25.268+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 4234 ms [2022-10-10T11:38:25.269+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:25.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:25.271+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:25.273+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:25.273+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:26.318+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 5284 ms [2022-10-10T11:38:26.318+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:26.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:26.322+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:26.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:26.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:27.368+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 6334 ms [2022-10-10T11:38:27.368+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:27.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:27.371+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:27.372+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:27.372+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:28.418+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 7384 ms [2022-10-10T11:38:28.421+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:28.421+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:28.425+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:28.426+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:28.426+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:29.468+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 8434 ms [2022-10-10T11:38:29.468+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:29.469+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:29.471+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:29.471+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:29.471+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:30.518+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 9484 ms [2022-10-10T11:38:30.519+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:30.519+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:30.522+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:30.522+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:30.523+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:31.568+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 10534 ms [2022-10-10T11:38:31.569+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:31.569+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:31.572+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:31.572+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:31.572+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:31.573+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 11, above threshold 10. [2022-10-10T11:38:32.618+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 11584 ms [2022-10-10T11:38:32.618+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:32.619+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:32.621+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:32.621+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:32.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:32.622+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 12, above threshold 10. [2022-10-10T11:38:33.668+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 12634 ms [2022-10-10T11:38:33.669+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:33.669+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:33.671+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:33.672+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:33.672+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:33.672+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 13, above threshold 10. [2022-10-10T11:38:34.718+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 13684 ms [2022-10-10T11:38:34.718+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:34.718+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:34.721+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:34.722+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:34.722+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:34.722+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 14, above threshold 10. [2022-10-10T11:38:35.768+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 14734 ms [2022-10-10T11:38:35.769+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:35.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:35.771+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:35.771+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:35.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:35.772+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 15, above threshold 10. [2022-10-10T11:38:36.092+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-10-10T11:38:36.093+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:38:36.093+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:38:36.098+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:36.098+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:36.098+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:38:36.818+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 15784 ms [2022-10-10T11:38:36.818+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:36.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:36.821+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:36.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:36.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:36.822+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 16, above threshold 10. [2022-10-10T11:38:37.868+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 16834 ms [2022-10-10T11:38:37.869+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:37.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:37.873+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:37.873+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:37.874+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:37.874+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 17, above threshold 10. [2022-10-10T11:38:38.918+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 17884 ms [2022-10-10T11:38:38.920+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:38.920+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:38.922+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:38.923+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:38.923+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:38.924+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 18, above threshold 10. [2022-10-10T11:38:39.968+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 18934 ms [2022-10-10T11:38:39.970+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:39.971+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:39.974+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:39.977+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:39.977+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:39.978+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 19, above threshold 10. [2022-10-10T11:38:41.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 19985 ms [2022-10-10T11:38:41.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:41.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:41.022+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:41.023+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:41.023+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:41.023+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 20, above threshold 10. [2022-10-10T11:38:42.071+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 21037 ms [2022-10-10T11:38:42.071+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:42.071+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:42.079+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:42.080+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:42.080+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:42.080+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 21, above threshold 10. [2022-10-10T11:38:43.118+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 22084 ms [2022-10-10T11:38:43.119+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:43.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:43.121+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:43.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:43.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:43.122+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 22, above threshold 10. [2022-10-10T11:38:44.168+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 23134 ms [2022-10-10T11:38:44.168+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:44.169+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:44.171+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:44.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:44.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:44.172+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23, above threshold 10. [2022-10-10T11:38:45.218+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 24184 ms [2022-10-10T11:38:45.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:45.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:45.224+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:45.225+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:45.227+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:45.227+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 24, above threshold 10. [2022-10-10T11:38:46.275+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 25241 ms [2022-10-10T11:38:46.275+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:46.275+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:46.283+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:46.284+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:46.284+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:46.284+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 25, above threshold 10. [2022-10-10T11:38:47.318+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 26284 ms [2022-10-10T11:38:47.318+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:47.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:47.321+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:47.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:47.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:47.322+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 26, above threshold 10. [2022-10-10T11:38:48.368+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 27334 ms [2022-10-10T11:38:48.369+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:48.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:48.376+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:48.378+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:48.379+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:48.379+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 27, above threshold 10. [2022-10-10T11:38:49.418+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 28384 ms [2022-10-10T11:38:49.418+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:49.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:49.421+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:49.422+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:49.422+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:49.422+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 28, above threshold 10. [2022-10-10T11:38:50.468+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 29434 ms [2022-10-10T11:38:50.468+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:50.469+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:50.471+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:50.472+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:50.472+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:50.472+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 29, above threshold 10. [2022-10-10T11:38:51.101+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-10-10T11:38:51.101+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:38:51.101+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:38:51.105+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:51.105+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:51.105+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:38:51.518+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 30484 ms [2022-10-10T11:38:51.519+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:51.519+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:51.541+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:51.542+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:51.542+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:51.542+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 30, above threshold 10. [2022-10-10T11:38:52.568+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 31534 ms [2022-10-10T11:38:52.569+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:52.569+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:52.571+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:52.572+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:52.572+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:52.572+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 31, above threshold 10. [2022-10-10T11:38:53.618+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 32584 ms [2022-10-10T11:38:53.619+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:53.619+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:53.623+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:53.624+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:53.624+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:53.624+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 32, above threshold 10. [2022-10-10T11:38:54.668+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 33634 ms [2022-10-10T11:38:54.668+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:54.669+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:54.671+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:54.671+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:54.671+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:54.671+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 33, above threshold 10. [2022-10-10T11:38:55.719+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 34685 ms [2022-10-10T11:38:55.719+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:55.719+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:55.722+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:55.723+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:55.723+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:55.723+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 34, above threshold 10. [2022-10-10T11:38:56.768+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 35734 ms [2022-10-10T11:38:56.768+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:56.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:56.771+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:56.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:56.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:56.772+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 35, above threshold 10. [2022-10-10T11:38:57.819+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 36784 ms [2022-10-10T11:38:57.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:57.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:57.822+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:57.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:57.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:57.822+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 36, above threshold 10. [2022-10-10T11:38:58.868+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 37834 ms [2022-10-10T11:38:58.868+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:58.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:58.870+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:58.871+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:58.871+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:58.871+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 37, above threshold 10. [2022-10-10T11:38:59.918+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 38884 ms [2022-10-10T11:38:59.919+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:38:59.919+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:38:59.921+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:38:59.922+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:38:59.923+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:38:59.923+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 38, above threshold 10. [2022-10-10T11:39:00.968+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 39934 ms [2022-10-10T11:39:00.968+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:00.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:00.972+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:00.974+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:00.974+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:00.974+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 39, above threshold 10. [2022-10-10T11:39:02.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 40985 ms [2022-10-10T11:39:02.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:02.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:02.027+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:02.028+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:02.028+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:02.028+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 40, above threshold 10. [2022-10-10T11:39:03.068+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 42034 ms [2022-10-10T11:39:03.068+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:03.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:03.071+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:03.072+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:03.072+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:03.072+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 41, above threshold 10. [2022-10-10T11:39:04.118+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 43084 ms [2022-10-10T11:39:04.118+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:04.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:04.122+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:04.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:04.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:04.122+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 42, above threshold 10. [2022-10-10T11:39:05.168+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 44134 ms [2022-10-10T11:39:05.168+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:05.169+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:05.171+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:05.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:05.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:05.172+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 43, above threshold 10. [2022-10-10T11:39:06.105+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-10-10T11:39:06.106+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:39:06.106+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:39:06.112+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:06.112+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:06.112+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:39:06.218+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 45184 ms [2022-10-10T11:39:06.218+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:06.218+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:06.221+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:06.221+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:06.221+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:06.221+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 44, above threshold 10. [2022-10-10T11:39:07.269+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 46234 ms [2022-10-10T11:39:07.269+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:07.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:07.273+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:07.274+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:07.274+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:07.274+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 45, above threshold 10. [2022-10-10T11:39:08.318+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 47284 ms [2022-10-10T11:39:08.318+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:08.318+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:08.322+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:08.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:08.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:08.322+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 46, above threshold 10. [2022-10-10T11:39:09.368+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 48334 ms [2022-10-10T11:39:09.368+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:09.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:09.372+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:09.372+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:09.372+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:09.372+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 47, above threshold 10. [2022-10-10T11:39:10.418+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 49384 ms [2022-10-10T11:39:10.418+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:10.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:10.422+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:10.422+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:10.422+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:10.422+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 48, above threshold 10. [2022-10-10T11:39:11.468+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 50434 ms [2022-10-10T11:39:11.469+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:11.469+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:11.472+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:11.473+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:11.473+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:11.473+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 49, above threshold 10. [2022-10-10T11:39:12.518+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 51484 ms [2022-10-10T11:39:12.519+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:12.519+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:12.522+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:12.522+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:12.523+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:12.523+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 50, above threshold 10. [2022-10-10T11:39:13.568+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 52534 ms [2022-10-10T11:39:13.568+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:13.568+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:13.570+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:13.571+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:13.571+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:13.571+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 51, above threshold 10. [2022-10-10T11:39:14.618+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 53584 ms [2022-10-10T11:39:14.618+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:14.618+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:14.621+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:14.621+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:14.621+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:14.621+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 52, above threshold 10. [2022-10-10T11:39:15.668+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 54634 ms [2022-10-10T11:39:15.668+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:15.669+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:15.671+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:15.671+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:15.671+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:15.671+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 53, above threshold 10. [2022-10-10T11:39:16.718+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 55684 ms [2022-10-10T11:39:16.719+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:16.719+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:16.726+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:16.727+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:16.727+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:16.727+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 54, above threshold 10. [2022-10-10T11:39:17.768+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 56734 ms [2022-10-10T11:39:17.769+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:17.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:17.771+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:17.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:17.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:17.772+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 55, above threshold 10. [2022-10-10T11:39:18.818+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 57784 ms [2022-10-10T11:39:18.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:18.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:18.821+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:18.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:18.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:18.823+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 56, above threshold 10. [2022-10-10T11:39:19.868+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 58834 ms [2022-10-10T11:39:19.869+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:19.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:19.871+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:19.871+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:19.871+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:19.872+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 57, above threshold 10. [2022-10-10T11:39:20.918+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 59884 ms [2022-10-10T11:39:20.918+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:20.919+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:20.921+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:20.922+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:20.922+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:20.922+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 58, above threshold 10. [2022-10-10T11:39:21.113+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-10-10T11:39:21.114+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:39:21.114+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:39:21.117+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:21.118+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:21.118+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:39:21.968+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 60934 ms [2022-10-10T11:39:21.968+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:21.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:21.974+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:21.974+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:21.974+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:21.974+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 59, above threshold 10. [2022-10-10T11:39:23.018+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 61984 ms [2022-10-10T11:39:23.018+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:23.018+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:23.021+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:23.021+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:23.021+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:23.021+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 60, above threshold 10. [2022-10-10T11:39:24.068+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 63034 ms [2022-10-10T11:39:24.068+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:24.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:24.071+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:24.072+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:24.072+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:24.072+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 61, above threshold 10. [2022-10-10T11:39:25.118+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 64084 ms [2022-10-10T11:39:25.119+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:25.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:25.121+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:25.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:25.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:25.122+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 62, above threshold 10. [2022-10-10T11:39:26.168+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 65134 ms [2022-10-10T11:39:26.168+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:26.169+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:26.171+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:26.171+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:26.171+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:26.171+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 63, above threshold 10. [2022-10-10T11:39:27.218+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 66184 ms [2022-10-10T11:39:27.218+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:27.218+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:27.221+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:27.221+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:27.221+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:27.221+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 64, above threshold 10. [2022-10-10T11:39:28.268+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 67234 ms [2022-10-10T11:39:28.271+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:28.271+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:28.273+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:28.274+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:28.274+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:28.274+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 65, above threshold 10. [2022-10-10T11:39:29.318+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 68284 ms [2022-10-10T11:39:29.319+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:29.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:29.322+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:29.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:29.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:29.322+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 66, above threshold 10. [2022-10-10T11:39:30.368+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 69334 ms [2022-10-10T11:39:30.368+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:30.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:30.371+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:30.371+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:30.371+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:30.371+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 67, above threshold 10. [2022-10-10T11:39:31.419+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 70384 ms [2022-10-10T11:39:31.419+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:31.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:31.421+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:31.422+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:31.422+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:31.422+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 68, above threshold 10. [2022-10-10T11:39:32.469+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 71435 ms [2022-10-10T11:39:32.469+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:32.469+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:32.472+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:32.472+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:32.472+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:32.472+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 69, above threshold 10. [2022-10-10T11:39:33.518+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 72484 ms [2022-10-10T11:39:33.518+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:33.518+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:33.521+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:33.521+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:33.521+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:33.521+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 70, above threshold 10. [2022-10-10T11:39:34.568+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 73534 ms [2022-10-10T11:39:34.568+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:34.568+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:34.571+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:34.571+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:34.571+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:34.571+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 71, above threshold 10. [2022-10-10T11:39:35.618+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 74584 ms [2022-10-10T11:39:35.618+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:35.618+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:35.621+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:35.621+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:35.621+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:35.621+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 72, above threshold 10. [2022-10-10T11:39:36.118+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-10-10T11:39:36.118+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:39:36.118+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:39:36.121+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:36.122+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:36.122+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:39:36.668+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 75634 ms [2022-10-10T11:39:36.668+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:36.669+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:36.671+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:36.671+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:36.671+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:36.671+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 73, above threshold 10. [2022-10-10T11:39:37.718+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 76684 ms [2022-10-10T11:39:37.718+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:37.718+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:37.720+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:37.721+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:37.721+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:37.721+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 74, above threshold 10. [2022-10-10T11:39:38.723+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 77689 ms [2022-10-10T11:39:38.723+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:38.723+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:38.726+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:38.726+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:38.726+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:38.726+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 75, above threshold 10. [2022-10-10T11:39:39.768+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 78734 ms [2022-10-10T11:39:39.768+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:39.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:39.772+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:39.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:39.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:39.772+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 76, above threshold 10. [2022-10-10T11:39:40.818+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 79784 ms [2022-10-10T11:39:40.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:40.820+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:40.822+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:40.823+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:40.823+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:40.823+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 77, above threshold 10. [2022-10-10T11:39:41.868+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 80834 ms [2022-10-10T11:39:41.869+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:41.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:41.871+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:41.872+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:41.872+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:41.872+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 78, above threshold 10. [2022-10-10T11:39:42.918+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 81884 ms [2022-10-10T11:39:42.918+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:42.918+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:42.920+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:42.921+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:42.921+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:42.921+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 79, above threshold 10. [2022-10-10T11:39:43.968+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 82934 ms [2022-10-10T11:39:43.968+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:43.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:43.971+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:43.971+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:43.971+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:43.971+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 80, above threshold 10. [2022-10-10T11:39:45.018+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 83984 ms [2022-10-10T11:39:45.018+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:45.018+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:45.020+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:45.021+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:45.021+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:45.021+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 81, above threshold 10. [2022-10-10T11:39:46.068+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 85034 ms [2022-10-10T11:39:46.068+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:46.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:46.071+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:46.072+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:46.072+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:46.072+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 82, above threshold 10. [2022-10-10T11:39:47.118+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 86084 ms [2022-10-10T11:39:47.118+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:47.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:47.121+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:47.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:47.122+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:47.122+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 83, above threshold 10. [2022-10-10T11:39:48.168+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 87134 ms [2022-10-10T11:39:48.169+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:48.169+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:48.172+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:48.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:48.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:48.172+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 84, above threshold 10. [2022-10-10T11:39:49.218+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 88184 ms [2022-10-10T11:39:49.218+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:49.218+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:49.221+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:49.221+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:49.221+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:49.221+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 85, above threshold 10. [2022-10-10T11:39:50.269+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 89235 ms [2022-10-10T11:39:50.269+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:50.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:50.272+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:50.273+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:50.273+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:50.273+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 86, above threshold 10. [2022-10-10T11:39:51.122+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-10-10T11:39:51.122+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:39:51.123+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:39:51.126+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:51.126+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:51.126+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:39:51.318+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 90284 ms [2022-10-10T11:39:51.319+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:51.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:51.324+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:51.324+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:51.325+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:51.325+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 87, above threshold 10. [2022-10-10T11:39:52.368+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 91334 ms [2022-10-10T11:39:52.368+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:52.368+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:52.373+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:52.373+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:52.373+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:52.373+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 88, above threshold 10. [2022-10-10T11:39:53.418+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 92384 ms [2022-10-10T11:39:53.418+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:53.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:53.421+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:53.421+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:53.422+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:53.422+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 89, above threshold 10. [2022-10-10T11:39:54.468+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 93434 ms [2022-10-10T11:39:54.468+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:54.468+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:54.471+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:54.471+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:54.471+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:54.471+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 90, above threshold 10. [2022-10-10T11:39:55.518+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 94484 ms [2022-10-10T11:39:55.518+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:55.518+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:55.521+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:55.521+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:55.521+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:55.521+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 91, above threshold 10. [2022-10-10T11:39:56.568+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 95534 ms [2022-10-10T11:39:56.568+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:56.569+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:56.571+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:56.572+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:56.572+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:56.572+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 92, above threshold 10. [2022-10-10T11:39:57.618+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 96584 ms [2022-10-10T11:39:57.619+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:57.619+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:57.621+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:57.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:57.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:57.622+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 93, above threshold 10. [2022-10-10T11:39:58.668+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 97634 ms [2022-10-10T11:39:58.668+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:58.671+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:58.673+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:58.674+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:58.674+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:58.674+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 94, above threshold 10. [2022-10-10T11:39:59.718+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 98684 ms [2022-10-10T11:39:59.718+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:39:59.718+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:39:59.721+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:39:59.722+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:39:59.722+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:39:59.722+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 95, above threshold 10. [2022-10-10T11:40:00.768+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 99734 ms [2022-10-10T11:40:00.768+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:00.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:00.771+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:00.771+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:00.771+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:00.771+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 96, above threshold 10. [2022-10-10T11:40:01.818+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 100784 ms [2022-10-10T11:40:01.818+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:01.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:01.821+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:01.821+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:01.821+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:01.821+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 97, above threshold 10. [2022-10-10T11:40:02.868+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 101834 ms [2022-10-10T11:40:02.869+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:02.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:02.872+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:02.872+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:02.872+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:02.872+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 98, above threshold 10. [2022-10-10T11:40:03.918+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 102884 ms [2022-10-10T11:40:03.919+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:03.919+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:03.921+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:03.922+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:03.922+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:03.922+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 99, above threshold 10. [2022-10-10T11:40:04.968+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 103934 ms [2022-10-10T11:40:04.969+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:04.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:04.971+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:04.972+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:04.972+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:04.972+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 100, above threshold 10. [2022-10-10T11:40:06.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 104985 ms [2022-10-10T11:40:06.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:06.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:06.022+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:06.022+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:06.022+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:06.022+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 101, above threshold 10. [2022-10-10T11:40:06.127+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-10-10T11:40:06.127+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:40:06.127+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:40:06.130+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:06.131+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:06.131+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:40:07.068+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 106034 ms [2022-10-10T11:40:07.068+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:07.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:07.071+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:07.071+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:07.071+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:07.071+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 102, above threshold 10. [2022-10-10T11:40:08.118+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 107084 ms [2022-10-10T11:40:08.118+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:08.118+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:08.121+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:08.121+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:08.121+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:08.121+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 103, above threshold 10. [2022-10-10T11:40:09.168+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 108134 ms [2022-10-10T11:40:09.168+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:09.169+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:09.171+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:09.171+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:09.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:09.172+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 104, above threshold 10. [2022-10-10T11:40:10.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 109185 ms [2022-10-10T11:40:10.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:10.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:10.223+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:10.223+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:10.224+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:10.224+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 105, above threshold 10. [2022-10-10T11:40:11.268+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 110234 ms [2022-10-10T11:40:11.268+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:11.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:11.271+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:11.271+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:11.271+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:11.271+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 106, above threshold 10. [2022-10-10T11:40:12.318+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 111284 ms [2022-10-10T11:40:12.319+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:12.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:12.321+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:12.321+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:12.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:12.322+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 107, above threshold 10. [2022-10-10T11:40:13.368+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 112334 ms [2022-10-10T11:40:13.368+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:13.368+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:13.371+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:13.372+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:13.372+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:13.372+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 108, above threshold 10. [2022-10-10T11:40:14.418+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 113384 ms [2022-10-10T11:40:14.418+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:14.418+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:14.421+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:14.421+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:14.421+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:14.421+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 109, above threshold 10. [2022-10-10T11:40:15.468+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 114434 ms [2022-10-10T11:40:15.468+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:15.469+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:15.473+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:15.474+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:15.474+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:15.474+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 110, above threshold 10. [2022-10-10T11:40:16.518+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 115484 ms [2022-10-10T11:40:16.519+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:16.520+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:16.524+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:16.525+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:16.525+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:16.525+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 111, above threshold 10. [2022-10-10T11:40:17.568+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 116534 ms [2022-10-10T11:40:17.569+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:17.569+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:17.572+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:17.572+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:17.572+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:17.572+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 112, above threshold 10. [2022-10-10T11:40:18.618+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 117584 ms [2022-10-10T11:40:18.619+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:18.619+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:18.622+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:18.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:18.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:18.622+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 113, above threshold 10. [2022-10-10T11:40:19.668+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 118634 ms [2022-10-10T11:40:19.668+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:19.669+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:19.671+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:19.671+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:19.671+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:19.671+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 114, above threshold 10. [2022-10-10T11:40:20.718+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 1 msgs to /events/POLICY-PDP-PAP. Oldest: 119684 ms [2022-10-10T11:40:20.721+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:20.721+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:20.724+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:20.724+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:20.724+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 1 message to send. [2022-10-10T11:40:20.724+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 115, above threshold 10. [2022-10-10T11:40:20.874+00:00|WARN|PdpMessageHandler|Timer-0] not registered: object:engine/apex/handler [2022-10-10T11:40:20.876+00:00|INFO|network|Timer-0] [OUT|DMAAP|POLICY-PDP-PAP] {"pdpType":"apex","state":"PASSIVE","healthy":"HEALTHY","description":"Pdp Heartbeat","statistics":{"pdpInstanceId":"apex-d89fe811-168c-4a33-8281-85b7ca1b5a33","timeStamp":"2022-10-10T11:40:20.874980Z","pdpGroupName":"defaultGroup","policyExecutedCount":0,"policyExecutedSuccessCount":0,"policyExecutedFailCount":0,"policyDeployCount":0,"policyDeploySuccessCount":0,"policyDeployFailCount":0,"policyUndeployCount":0,"policyUndeploySuccessCount":0,"policyUndeployFailCount":0},"messageName":"PDP_STATUS","requestId":"4ee310c5-5690-4727-a120-05253fc0db10","timestampMs":1665402020874,"name":"apex-d89fe811-168c-4a33-8281-85b7ca1b5a33","pdpGroup":"defaultGroup"} [2022-10-10T11:40:21.131+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-10-10T11:40:21.132+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:40:21.132+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:40:21.140+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:21.141+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:21.141+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:40:21.768+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 120734 ms [2022-10-10T11:40:21.768+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:21.768+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:21.771+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:21.771+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:21.771+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:21.771+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 116, above threshold 10. [2022-10-10T11:40:22.771+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 121737 ms [2022-10-10T11:40:22.771+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:22.771+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:22.779+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:22.779+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:22.779+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:22.780+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 117, above threshold 10. [2022-10-10T11:40:23.818+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 122784 ms [2022-10-10T11:40:23.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:23.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:23.822+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:23.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:23.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:23.822+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 118, above threshold 10. [2022-10-10T11:40:24.868+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 123834 ms [2022-10-10T11:40:24.869+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:24.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:24.872+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:24.873+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:24.873+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:24.873+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 119, above threshold 10. [2022-10-10T11:40:25.918+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 124884 ms [2022-10-10T11:40:25.919+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:25.919+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:25.921+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:25.921+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:25.922+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:25.922+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 120, above threshold 10. [2022-10-10T11:40:26.968+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 125934 ms [2022-10-10T11:40:26.969+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:26.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:26.971+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:26.971+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:26.972+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:26.972+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 121, above threshold 10. [2022-10-10T11:40:28.018+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 126984 ms [2022-10-10T11:40:28.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:28.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:28.021+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:28.021+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:28.021+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:28.021+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 122, above threshold 10. [2022-10-10T11:40:29.068+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 128034 ms [2022-10-10T11:40:29.068+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:29.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:29.071+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:29.072+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:29.072+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:29.072+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 123, above threshold 10. [2022-10-10T11:40:30.118+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 129084 ms [2022-10-10T11:40:30.118+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:30.118+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:30.121+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:30.121+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:30.121+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:30.121+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 124, above threshold 10. [2022-10-10T11:40:31.168+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 130134 ms [2022-10-10T11:40:31.168+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:31.168+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:31.171+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:31.171+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:31.171+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:31.171+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 125, above threshold 10. [2022-10-10T11:40:32.218+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 131184 ms [2022-10-10T11:40:32.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:32.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:32.221+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:32.222+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:32.222+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:32.222+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 126, above threshold 10. [2022-10-10T11:40:33.268+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 132234 ms [2022-10-10T11:40:33.269+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:33.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:33.273+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:33.273+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:33.274+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:33.274+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 127, above threshold 10. [2022-10-10T11:40:34.318+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 133284 ms [2022-10-10T11:40:34.318+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:34.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:34.321+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:34.321+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:34.321+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:34.321+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 128, above threshold 10. [2022-10-10T11:40:35.369+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 134335 ms [2022-10-10T11:40:35.370+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:35.370+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:35.373+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:35.373+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:35.373+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:35.373+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 129, above threshold 10. [2022-10-10T11:40:36.141+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-10-10T11:40:36.141+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:40:36.142+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:40:36.144+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:36.145+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:36.145+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:40:36.418+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 135384 ms [2022-10-10T11:40:36.418+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:36.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:36.421+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:36.421+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:36.421+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:36.421+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 130, above threshold 10. [2022-10-10T11:40:37.468+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 136434 ms [2022-10-10T11:40:37.469+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:37.469+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:37.471+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:37.471+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:37.472+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:37.472+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 131, above threshold 10. [2022-10-10T11:40:38.518+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 137484 ms [2022-10-10T11:40:38.518+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:38.519+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:38.520+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:38.521+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:38.521+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:38.521+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 132, above threshold 10. [2022-10-10T11:40:39.568+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 138534 ms [2022-10-10T11:40:39.568+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:39.568+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:39.571+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:39.571+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:39.571+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:39.571+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 133, above threshold 10. [2022-10-10T11:40:40.618+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 139584 ms [2022-10-10T11:40:40.619+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:40.619+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:40.621+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:40.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:40.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:40.622+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 134, above threshold 10. [2022-10-10T11:40:41.668+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 140634 ms [2022-10-10T11:40:41.668+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:41.669+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:41.671+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:41.671+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:41.671+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:41.671+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 135, above threshold 10. [2022-10-10T11:40:42.718+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 141684 ms [2022-10-10T11:40:42.718+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:42.719+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:42.721+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:42.721+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:42.721+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:42.721+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 136, above threshold 10. [2022-10-10T11:40:43.768+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 142734 ms [2022-10-10T11:40:43.768+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:43.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:43.771+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:43.771+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:43.771+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:43.771+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 137, above threshold 10. [2022-10-10T11:40:44.818+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 143784 ms [2022-10-10T11:40:44.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:44.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:44.822+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:44.823+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:44.823+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:44.823+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 138, above threshold 10. [2022-10-10T11:40:45.868+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 144834 ms [2022-10-10T11:40:45.868+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:45.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:45.871+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:45.871+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:45.871+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:45.871+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 139, above threshold 10. [2022-10-10T11:40:46.918+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 145884 ms [2022-10-10T11:40:46.919+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:46.919+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:46.922+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:46.923+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:46.923+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:46.923+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 140, above threshold 10. [2022-10-10T11:40:47.968+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 146934 ms [2022-10-10T11:40:47.969+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:47.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:47.972+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:47.973+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:47.973+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:47.973+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 141, above threshold 10. [2022-10-10T11:40:49.018+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 147984 ms [2022-10-10T11:40:49.018+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:49.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:49.021+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:49.021+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:49.021+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:49.021+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 142, above threshold 10. [2022-10-10T11:40:50.068+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 149034 ms [2022-10-10T11:40:50.068+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:50.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:50.071+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:50.071+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:50.071+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:50.071+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 143, above threshold 10. [2022-10-10T11:40:51.118+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 150084 ms [2022-10-10T11:40:51.118+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:51.121+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:51.123+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:51.124+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:51.124+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:51.124+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 144, above threshold 10. [2022-10-10T11:40:51.145+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-10-10T11:40:51.145+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:40:51.146+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:40:51.148+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:51.148+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:51.149+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:40:52.168+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 151134 ms [2022-10-10T11:40:52.168+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:52.168+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:52.171+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:52.171+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:52.171+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:52.171+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 145, above threshold 10. [2022-10-10T11:40:53.218+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 152184 ms [2022-10-10T11:40:53.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:53.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:53.223+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:53.223+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:53.223+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:53.223+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 146, above threshold 10. [2022-10-10T11:40:54.268+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 153234 ms [2022-10-10T11:40:54.269+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:54.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:54.272+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:54.272+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:54.273+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:54.273+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 147, above threshold 10. [2022-10-10T11:40:55.318+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 154284 ms [2022-10-10T11:40:55.319+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:55.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:55.322+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:55.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:55.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:55.322+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 148, above threshold 10. [2022-10-10T11:40:56.368+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 155334 ms [2022-10-10T11:40:56.369+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:56.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:56.371+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:56.371+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:56.371+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:56.371+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 149, above threshold 10. [2022-10-10T11:40:57.418+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 156384 ms [2022-10-10T11:40:57.418+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:57.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:57.420+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:57.421+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:57.421+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:57.421+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 150, above threshold 10. [2022-10-10T11:40:58.468+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 157434 ms [2022-10-10T11:40:58.468+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:58.468+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:58.470+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:58.471+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:58.471+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:58.471+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 151, above threshold 10. [2022-10-10T11:40:59.518+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 158484 ms [2022-10-10T11:40:59.518+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:40:59.518+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:40:59.520+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:40:59.521+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:40:59.521+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:40:59.521+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 152, above threshold 10. [2022-10-10T11:41:00.568+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 159534 ms [2022-10-10T11:41:00.568+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:00.568+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:00.570+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:00.571+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:00.571+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:00.571+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 153, above threshold 10. [2022-10-10T11:41:01.618+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 160584 ms [2022-10-10T11:41:01.618+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:01.619+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:01.621+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:01.621+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:01.621+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:01.621+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 154, above threshold 10. [2022-10-10T11:41:02.668+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 161634 ms [2022-10-10T11:41:02.668+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:02.668+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:02.671+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:02.671+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:02.671+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:02.671+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 155, above threshold 10. [2022-10-10T11:41:03.718+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 162684 ms [2022-10-10T11:41:03.719+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:03.719+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:03.722+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:03.723+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:03.723+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:03.723+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 156, above threshold 10. [2022-10-10T11:41:04.768+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 163734 ms [2022-10-10T11:41:04.769+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:04.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:04.772+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:04.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:04.772+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:04.772+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 157, above threshold 10. [2022-10-10T11:41:05.818+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 164784 ms [2022-10-10T11:41:05.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:05.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:05.821+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:05.821+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:05.821+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:05.821+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 158, above threshold 10. [2022-10-10T11:41:06.149+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-10-10T11:41:06.149+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:41:06.149+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:41:06.152+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:06.152+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:06.152+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:41:06.868+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 165834 ms [2022-10-10T11:41:06.868+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:06.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:06.871+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:06.871+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:06.871+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:06.871+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 159, above threshold 10. [2022-10-10T11:41:07.918+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 166884 ms [2022-10-10T11:41:07.919+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:07.919+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:07.921+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:07.921+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:07.922+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:07.922+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 160, above threshold 10. [2022-10-10T11:41:08.968+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 167934 ms [2022-10-10T11:41:08.969+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:08.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:08.972+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:08.973+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:08.973+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:08.973+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 161, above threshold 10. [2022-10-10T11:41:10.018+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 168984 ms [2022-10-10T11:41:10.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:10.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:10.021+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:10.022+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:10.022+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:10.022+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 162, above threshold 10. [2022-10-10T11:41:11.070+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 170036 ms [2022-10-10T11:41:11.070+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:11.070+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:11.073+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:11.073+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:11.073+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:11.073+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 163, above threshold 10. [2022-10-10T11:41:12.118+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 171084 ms [2022-10-10T11:41:12.118+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:12.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:12.121+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:12.121+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:12.121+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:12.121+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 164, above threshold 10. [2022-10-10T11:41:13.168+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 172134 ms [2022-10-10T11:41:13.168+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:13.168+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:13.170+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:13.171+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:13.171+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:13.171+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 165, above threshold 10. [2022-10-10T11:41:14.218+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 173184 ms [2022-10-10T11:41:14.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:14.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:14.221+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:14.221+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:14.221+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:14.221+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 166, above threshold 10. [2022-10-10T11:41:15.268+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 174234 ms [2022-10-10T11:41:15.269+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:15.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:15.272+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:15.272+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:15.272+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:15.272+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 167, above threshold 10. [2022-10-10T11:41:16.318+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 175284 ms [2022-10-10T11:41:16.318+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:16.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:16.320+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:16.321+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:16.321+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:16.321+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 168, above threshold 10. [2022-10-10T11:41:17.368+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 176334 ms [2022-10-10T11:41:17.368+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:17.368+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:17.371+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:17.371+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:17.371+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:17.371+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 169, above threshold 10. [2022-10-10T11:41:18.418+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 177384 ms [2022-10-10T11:41:18.418+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:18.418+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:18.421+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:18.421+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:18.421+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:18.421+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 170, above threshold 10. [2022-10-10T11:41:19.468+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 178434 ms [2022-10-10T11:41:19.468+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:19.468+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:19.470+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:19.471+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:19.471+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:19.471+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 171, above threshold 10. [2022-10-10T11:41:20.518+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 179484 ms [2022-10-10T11:41:20.518+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:20.518+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:20.521+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:20.521+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:20.521+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:20.521+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 172, above threshold 10. [2022-10-10T11:41:21.153+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-10-10T11:41:21.153+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:41:21.153+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:41:21.156+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:21.157+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:21.157+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:41:21.568+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 180534 ms [2022-10-10T11:41:21.568+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:21.569+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:21.571+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:21.571+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:21.571+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:21.571+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 173, above threshold 10. [2022-10-10T11:41:22.618+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 181584 ms [2022-10-10T11:41:22.619+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:22.619+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:22.621+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:22.621+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:22.621+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:22.621+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 174, above threshold 10. [2022-10-10T11:41:23.668+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 182634 ms [2022-10-10T11:41:23.668+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:23.668+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:23.672+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:23.673+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:23.673+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:23.673+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 175, above threshold 10. [2022-10-10T11:41:24.718+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 183684 ms [2022-10-10T11:41:24.718+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:24.719+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:24.720+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:24.721+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:24.721+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:24.721+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 176, above threshold 10. [2022-10-10T11:41:25.768+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 184734 ms [2022-10-10T11:41:25.768+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:25.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:25.771+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:25.771+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:25.771+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:25.771+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 177, above threshold 10. [2022-10-10T11:41:26.818+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 185784 ms [2022-10-10T11:41:26.819+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:26.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:26.821+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:26.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:26.822+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:26.822+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 178, above threshold 10. [2022-10-10T11:41:27.868+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 186834 ms [2022-10-10T11:41:27.868+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:27.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:27.871+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:27.871+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:27.872+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:27.872+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 179, above threshold 10. [2022-10-10T11:41:28.918+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 187884 ms [2022-10-10T11:41:28.918+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:28.919+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:28.923+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:28.923+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:28.923+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:28.924+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 180, above threshold 10. [2022-10-10T11:41:29.968+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 188934 ms [2022-10-10T11:41:29.969+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:29.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:29.971+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:29.972+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:29.972+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:29.972+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 181, above threshold 10. [2022-10-10T11:41:31.018+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 189984 ms [2022-10-10T11:41:31.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:31.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:31.021+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:31.022+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:31.022+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:31.022+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 182, above threshold 10. [2022-10-10T11:41:32.068+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 191034 ms [2022-10-10T11:41:32.069+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:32.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:32.072+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:32.072+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:32.072+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:32.072+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 183, above threshold 10. [2022-10-10T11:41:33.118+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 192084 ms [2022-10-10T11:41:33.118+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:33.118+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:33.121+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:33.121+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:33.121+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:33.121+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 184, above threshold 10. [2022-10-10T11:41:34.168+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 193134 ms [2022-10-10T11:41:34.168+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:34.168+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:34.171+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:34.171+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:34.171+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:34.171+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 185, above threshold 10. [2022-10-10T11:41:35.218+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 194184 ms [2022-10-10T11:41:35.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:35.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:35.229+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:35.230+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:35.230+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:35.230+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 186, above threshold 10. [2022-10-10T11:41:36.157+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-10-10T11:41:36.157+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:41:36.157+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:41:36.160+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:36.161+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:36.161+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:41:36.268+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 195234 ms [2022-10-10T11:41:36.268+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:36.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:36.271+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:36.271+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:36.271+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:36.271+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 187, above threshold 10. [2022-10-10T11:41:37.318+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 196284 ms [2022-10-10T11:41:37.319+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:37.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:37.321+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:37.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:37.322+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:37.322+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 188, above threshold 10. [2022-10-10T11:41:38.368+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 197334 ms [2022-10-10T11:41:38.368+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:38.368+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:38.370+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:38.371+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:38.371+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:38.371+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 189, above threshold 10. [2022-10-10T11:41:39.418+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 198384 ms [2022-10-10T11:41:39.418+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:39.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:39.421+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:39.421+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:39.421+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:39.421+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 190, above threshold 10. [2022-10-10T11:41:40.468+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 199434 ms [2022-10-10T11:41:40.468+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:40.469+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:40.472+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:40.474+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:40.474+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:40.474+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 191, above threshold 10. [2022-10-10T11:41:41.518+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 200484 ms [2022-10-10T11:41:41.518+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:41.519+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:41.521+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:41.521+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:41.522+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:41.522+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 192, above threshold 10. [2022-10-10T11:41:42.568+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 201534 ms [2022-10-10T11:41:42.569+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:42.569+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:42.571+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:42.572+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:42.572+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:42.572+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 193, above threshold 10. [2022-10-10T11:41:43.618+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 202584 ms [2022-10-10T11:41:43.618+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:43.618+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:43.620+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:43.621+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:43.621+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:43.621+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 194, above threshold 10. [2022-10-10T11:41:44.668+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 203634 ms [2022-10-10T11:41:44.668+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:44.669+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:44.671+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:44.672+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:44.672+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:44.672+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 195, above threshold 10. [2022-10-10T11:41:45.718+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 204684 ms [2022-10-10T11:41:45.718+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:45.718+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:45.721+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:45.721+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:45.721+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:45.721+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 196, above threshold 10. [2022-10-10T11:41:46.768+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 205734 ms [2022-10-10T11:41:46.768+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:46.768+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:46.771+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:46.771+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:46.771+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:46.771+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 197, above threshold 10. [2022-10-10T11:41:47.771+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 206737 ms [2022-10-10T11:41:47.771+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:47.772+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:47.776+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:47.776+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:47.777+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:47.777+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 198, above threshold 10. [2022-10-10T11:41:48.818+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 207784 ms [2022-10-10T11:41:48.818+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:48.818+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:48.821+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:48.821+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:48.821+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:48.821+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 199, above threshold 10. [2022-10-10T11:41:49.868+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 208834 ms [2022-10-10T11:41:49.869+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:49.869+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:49.871+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:49.872+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:49.872+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:49.872+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 200, above threshold 10. [2022-10-10T11:41:50.918+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 209884 ms [2022-10-10T11:41:50.919+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:50.919+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:50.922+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:50.924+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:50.924+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:50.925+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 201, above threshold 10. [2022-10-10T11:41:51.161+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-10-10T11:41:51.162+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:41:51.162+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:41:51.165+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:51.165+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:51.165+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:41:51.968+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 210934 ms [2022-10-10T11:41:51.969+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:51.969+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:51.972+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:51.972+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:51.972+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:51.972+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 202, above threshold 10. [2022-10-10T11:41:53.018+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 211984 ms [2022-10-10T11:41:53.019+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:53.019+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:53.022+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:53.024+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:53.025+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:53.025+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 203, above threshold 10. [2022-10-10T11:41:54.068+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 213034 ms [2022-10-10T11:41:54.069+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:54.069+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:54.076+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:54.078+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:54.078+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:54.078+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 204, above threshold 10. [2022-10-10T11:41:55.118+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 214084 ms [2022-10-10T11:41:55.118+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:55.119+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:55.122+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:55.123+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:55.123+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:55.123+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 205, above threshold 10. [2022-10-10T11:41:56.168+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 215134 ms [2022-10-10T11:41:56.169+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:56.169+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:56.172+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:56.172+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:56.174+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:56.174+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 206, above threshold 10. [2022-10-10T11:41:57.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 216185 ms [2022-10-10T11:41:57.219+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:57.219+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:57.221+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:57.222+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:57.222+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:57.222+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 207, above threshold 10. [2022-10-10T11:41:58.268+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 217234 ms [2022-10-10T11:41:58.268+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:58.269+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:58.271+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:58.271+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:58.271+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:58.272+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 208, above threshold 10. [2022-10-10T11:41:59.318+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 218284 ms [2022-10-10T11:41:59.318+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:41:59.319+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:41:59.321+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:41:59.321+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:41:59.321+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:41:59.321+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 209, above threshold 10. [2022-10-10T11:42:00.368+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 219334 ms [2022-10-10T11:42:00.369+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:42:00.369+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:42:00.371+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:42:00.372+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:42:00.372+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:42:00.372+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 210, above threshold 10. [2022-10-10T11:42:01.418+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 220384 ms [2022-10-10T11:42:01.419+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:42:01.419+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:42:01.420+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:42:01.421+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:42:01.421+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:42:01.421+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 211, above threshold 10. [2022-10-10T11:42:02.468+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 221434 ms [2022-10-10T11:42:02.468+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:42:02.469+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:42:02.471+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:42:02.471+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:42:02.471+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:42:02.471+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 212, above threshold 10. [2022-10-10T11:42:03.518+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 222484 ms [2022-10-10T11:42:03.518+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:42:03.518+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:42:03.520+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:42:03.521+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:42:03.521+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:42:03.521+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 213, above threshold 10. [2022-10-10T11:42:04.568+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 223534 ms [2022-10-10T11:42:04.568+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:42:04.569+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:42:04.571+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:42:04.571+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:42:04.571+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:42:04.571+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 214, above threshold 10. [2022-10-10T11:42:05.619+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 224585 ms [2022-10-10T11:42:05.620+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:42:05.620+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:42:05.622+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:42:05.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:42:05.622+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:42:05.622+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 215, above threshold 10. [2022-10-10T11:42:06.165+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-PDP-PAP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=a61bd33b-f311-45cb-bd9e-2ad9cca10be3, consumerInstance=onap-policy-apex-pdp-0, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:123) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) ... 5 common frames omitted [2022-10-10T11:42:06.166+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-PDP-PAP] UEB GET /events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 [2022-10-10T11:42:06.166+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] GET http://message-router:3904/events/POLICY-PDP-PAP/a61bd33b-f311-45cb-bd9e-2ad9cca10be3/onap-policy-apex-pdp-0?timeout=15000 (anonymous) ... [2022-10-10T11:42:06.169+00:00|INFO|HttpClient|DMAAP-source-POLICY-PDP-PAP] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:42:06.169+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable [2022-10-10T11:42:06.169+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-PDP-PAP] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2022-10-10T11:42:06.668+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 225634 ms [2022-10-10T11:42:06.668+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:42:06.669+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:42:06.670+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:42:06.671+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:42:06.671+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:42:06.671+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 216, above threshold 10. [2022-10-10T11:42:07.718+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 226684 ms [2022-10-10T11:42:07.719+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:42:07.719+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:42:07.721+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:42:07.722+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:42:07.722+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:42:07.722+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 217, above threshold 10. [2022-10-10T11:42:08.768+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 227734 ms [2022-10-10T11:42:08.768+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:42:08.769+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:42:08.772+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:42:08.773+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:42:08.773+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:42:08.773+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 218, above threshold 10. [2022-10-10T11:42:09.818+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 228784 ms [2022-10-10T11:42:09.818+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:42:09.819+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:42:09.821+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:42:09.821+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:42:09.821+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:42:09.821+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 219, above threshold 10. [2022-10-10T11:42:10.868+00:00|INFO|CambriaSimplerBatchPublisher|pool-2-thread-1] sending 2 msgs to /events/POLICY-PDP-PAP. Oldest: 229834 ms [2022-10-10T11:42:10.868+00:00|WARN|HostSelector|pool-2-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2022-10-10T11:42:10.868+00:00|INFO|HttpClient|pool-2-thread-1] POST http://message-router:3904/events/POLICY-PDP-PAP (anonymous) ... [2022-10-10T11:42:10.871+00:00|INFO|HttpClient|pool-2-thread-1] --> HTTP/1.1 503 Service Unavailable [2022-10-10T11:42:10.871+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] HTTP/1.1 503 Service Unavailable com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:816) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-10-10T11:42:10.871+00:00|WARN|CambriaSimplerBatchPublisher|pool-2-thread-1] Send failed, 2 message to send. [2022-10-10T11:42:10.871+00:00|ERROR|CambriaSimplerBatchPublisher|pool-2-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 220, above threshold 10. [2022-10-10T1