[2023-03-11T20:04:32.686+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:32.686+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23025, above threshold 10. [2023-03-11T20:04:33.719+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59959732 ms [2023-03-11T20:04:33.721+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:33.721+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:33.726+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:33.726+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:33.731+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:04:33.731+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:33.734+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:33.734+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:33.746+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:33.747+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:33.747+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:33.747+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23026, above threshold 10. [2023-03-11T20:04:34.769+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59960782 ms [2023-03-11T20:04:34.770+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:34.770+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:34.776+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:34.777+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:34.787+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:34.788+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:34.792+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:34.792+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:34.817+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:34.817+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:34.817+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:34.817+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23027, above threshold 10. [2023-03-11T20:04:35.819+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59961832 ms [2023-03-11T20:04:35.820+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:35.820+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:35.830+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:35.830+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:35.834+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:35.834+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:35.837+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:35.837+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:35.840+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:35.840+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:35.840+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:35.840+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23028, above threshold 10. [2023-03-11T20:04:36.873+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59962886 ms [2023-03-11T20:04:36.874+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:36.874+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:36.883+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:36.883+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:36.891+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:36.891+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:36.900+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:36.900+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:36.912+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Broken pipe (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:36.913+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Broken pipe (Write failed) java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:36.913+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:36.913+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23029, above threshold 10. [2023-03-11T20:04:37.919+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59963932 ms [2023-03-11T20:04:37.920+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:37.920+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:37.925+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:37.925+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:37.930+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:37.930+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:37.933+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:37.933+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:37.937+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:37.937+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:37.937+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:37.937+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23030, above threshold 10. [2023-03-11T20:04:38.969+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59964982 ms [2023-03-11T20:04:38.970+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:38.970+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:38.977+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:38.977+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:38.984+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:38.984+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:38.987+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:38.987+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:38.990+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:38.990+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:38.990+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:38.990+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23031, above threshold 10. [2023-03-11T20:04:39.110+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:39.110+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:04:39.110+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:39.110+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:04:39.112+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:04:39.113+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:39.115+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router:3904: The target server failed to respond [2023-03-11T20:04:39.115+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:39.118+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:04:39.118+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:39.121+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:04:39.121+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:04:39.121+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:04:40.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59966032 ms [2023-03-11T20:04:40.020+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:40.020+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:40.024+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:40.024+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:40.026+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:40.026+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:40.028+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:40.028+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:40.030+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:40.031+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:40.031+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:40.031+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23032, above threshold 10. [2023-03-11T20:04:41.069+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59967082 ms [2023-03-11T20:04:41.070+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:41.070+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:41.074+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:41.074+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:41.076+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:41.077+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:41.079+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:04:41.079+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:41.082+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:41.082+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:41.082+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:41.082+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23033, above threshold 10. [2023-03-11T20:04:42.119+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59968132 ms [2023-03-11T20:04:42.120+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:42.120+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:42.125+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:42.125+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:42.128+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:42.128+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:42.137+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:42.137+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:42.148+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:42.148+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:42.148+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:42.148+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23034, above threshold 10. [2023-03-11T20:04:43.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59969182 ms [2023-03-11T20:04:43.170+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:43.170+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:43.174+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:43.174+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:43.176+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:43.176+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:43.179+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:43.180+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:43.182+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:43.182+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:43.182+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:43.182+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23035, above threshold 10. [2023-03-11T20:04:44.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59970232 ms [2023-03-11T20:04:44.220+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:44.220+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:44.224+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:44.224+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:44.227+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:44.227+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:44.232+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:44.232+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:44.236+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:44.236+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:44.237+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:44.237+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23036, above threshold 10. [2023-03-11T20:04:45.269+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59971282 ms [2023-03-11T20:04:45.270+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:45.270+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:45.274+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:45.274+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:45.277+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:45.277+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:45.279+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:45.279+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:45.283+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:45.284+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:45.284+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:45.284+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23037, above threshold 10. [2023-03-11T20:04:46.319+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59972332 ms [2023-03-11T20:04:46.320+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:46.320+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:46.326+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:46.326+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:46.347+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:46.347+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:46.364+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:46.364+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:46.367+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:46.367+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:46.368+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:46.368+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23038, above threshold 10. [2023-03-11T20:04:47.369+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59973382 ms [2023-03-11T20:04:47.370+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:47.370+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:47.375+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:47.375+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:47.379+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:47.379+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:47.388+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:47.388+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:47.395+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:47.395+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:47.395+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:47.395+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23039, above threshold 10. [2023-03-11T20:04:48.419+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59974432 ms [2023-03-11T20:04:48.420+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:48.420+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:48.427+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:48.427+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:48.451+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:48.451+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:48.476+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:48.476+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:48.485+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:48.485+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:48.485+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:48.485+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23040, above threshold 10. [2023-03-11T20:04:49.519+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59975532 ms [2023-03-11T20:04:49.520+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:49.520+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:49.525+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:49.525+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:49.539+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:49.539+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:49.544+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:49.544+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:49.555+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:49.555+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:49.555+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:49.555+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23041, above threshold 10. [2023-03-11T20:04:50.569+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59976582 ms [2023-03-11T20:04:50.570+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:50.570+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:50.578+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:04:50.578+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:50.593+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:50.593+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:50.605+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:50.605+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:50.611+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:50.611+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:50.611+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:50.611+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23042, above threshold 10. [2023-03-11T20:04:51.619+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59977632 ms [2023-03-11T20:04:51.620+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:51.620+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:51.625+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:51.625+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:51.637+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:51.637+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:51.647+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:51.647+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:51.650+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:51.650+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:51.650+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:51.651+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23043, above threshold 10. [2023-03-11T20:04:52.669+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59978682 ms [2023-03-11T20:04:52.670+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:52.670+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:52.684+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:52.684+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:52.692+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:52.692+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:52.700+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:52.700+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:52.703+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:52.703+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:52.703+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:52.703+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23044, above threshold 10. [2023-03-11T20:04:53.719+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59979732 ms [2023-03-11T20:04:53.720+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:53.720+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:53.723+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:53.723+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:53.735+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:53.735+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:53.739+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:53.739+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:53.745+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:53.745+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:53.745+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:53.745+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23045, above threshold 10. [2023-03-11T20:04:54.122+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:54.122+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:04:54.122+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:54.122+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:04:54.129+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:04:54.129+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:54.132+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:04:54.132+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:54.135+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:04:54.135+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:54.139+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. message-router:3904 failed to respond; blacklisting for 2 minutes [2023-03-11T20:04:54.139+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of message-router:3904 failed to respond [2023-03-11T20:04:54.139+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:04:54.769+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59980782 ms [2023-03-11T20:04:54.770+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:54.770+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:54.780+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:54.780+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:54.791+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:54.792+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:54.794+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:04:54.794+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:54.809+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:54.809+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:54.809+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:54.809+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23046, above threshold 10. [2023-03-11T20:04:55.819+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59981832 ms [2023-03-11T20:04:55.820+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:55.820+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:55.827+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:55.827+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:55.834+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:55.834+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:55.863+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:55.863+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:55.880+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:55.880+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:55.880+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:55.880+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23047, above threshold 10. [2023-03-11T20:04:56.919+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59982932 ms [2023-03-11T20:04:56.920+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:56.920+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:56.923+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:56.923+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:56.925+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:56.925+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:56.928+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:56.928+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:56.931+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:56.931+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:56.931+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:56.931+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23048, above threshold 10. [2023-03-11T20:04:57.969+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59983982 ms [2023-03-11T20:04:57.970+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:57.970+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:04:57.979+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:57.979+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:57.983+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:57.983+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:57.989+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:04:57.989+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:04:57.994+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:04:57.995+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:04:57.995+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:04:57.995+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23049, above threshold 10. [2023-03-11T20:04:59.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 500 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 59985032 ms [2023-03-11T20:04:59.020+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:04:59.020+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:05:09.140+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch org.apache.http.NoHttpResponseException: message-router:3904 failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:05:09.140+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:05:09.140+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:05:09.140+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:05:09.145+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:05:09.145+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:05:09.151+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:05:09.151+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:05:09.154+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:05:09.154+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:05:09.158+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:05:09.158+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:05:09.158+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:05:11.050+00:00|INFO|network|pool-3-thread-1] [OUT|DMAAP|POLICY-ACRUNTIME-PARTICIPANT] {"state":"UNKNOWN","healthStatus":"UNKNOWN","participantStatistics":{"participantId":{"name":"K8sParticipant0","version":"1.0.0"},"timeStamp":"2023-03-11T20:05:11.049723Z","state":"UNKNOWN","healthStatus":"UNKNOWN","eventCount":0,"lastExecutionTime":0,"averageExecutionTime":0.0,"upTime":0,"lastEnterTime":0,"lastStart":0},"participantDefinitionUpdates":[],"automationCompositionInfoList":[],"messageType":"PARTICIPANT_STATUS","messageId":"f82f1b3e-c1ca-4517-b516-a5b5bcacd88d","timestamp":"2023-03-11T20:05:11.049783Z","participantType":{"name":"org.onap.policy.clamp.acm.KubernetesParticipant","version":"2.3.4"},"participantId":{"name":"K8sParticipant0","version":"1.0.0"}} [2023-03-11T20:05:24.158+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:05:24.158+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:05:24.158+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:05:24.158+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:05:24.164+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:05:24.164+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:05:24.167+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:05:24.167+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:05:24.170+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:05:24.170+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:05:24.173+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:05:24.173+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:05:24.173+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:05:39.173+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:05:39.174+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:05:39.174+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:05:39.174+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:05:39.178+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:05:39.178+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:05:39.180+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:05:39.180+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:05:39.183+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:05:39.183+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:05:39.185+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:05:39.185+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:05:39.185+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:05:54.186+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:05:54.186+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:05:54.186+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:05:54.186+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:05:54.200+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:05:54.200+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:05:54.204+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:05:54.204+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:05:54.206+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:05:54.206+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:05:54.209+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:05:54.209+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:05:54.209+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:06:08.100+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:06:08.100+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:08.120+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:06:08.120+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:08.139+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:06:08.139+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:08.147+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:06:08.148+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:06:08.148+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 500 message to send. [2023-03-11T20:06:08.148+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23050, above threshold 10. [2023-03-11T20:06:09.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 501 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60055182 ms [2023-03-11T20:06:09.170+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:06:09.170+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:06:09.180+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:06:09.180+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:09.187+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:06:09.187+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:09.190+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:06:09.190+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:09.192+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:06:09.192+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:06:09.193+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 501 message to send. [2023-03-11T20:06:09.193+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23051, above threshold 10. [2023-03-11T20:06:09.209+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:06:09.209+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:06:09.209+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:06:09.209+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:06:09.212+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:06:09.212+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:09.214+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:06:09.215+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:09.217+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:06:09.217+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:09.219+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. message-router:3904 failed to respond; blacklisting for 2 minutes [2023-03-11T20:06:09.219+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of message-router:3904 failed to respond [2023-03-11T20:06:09.219+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:06:10.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 501 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60056232 ms [2023-03-11T20:06:10.220+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:06:10.220+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:06:10.223+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:06:10.223+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:10.225+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:06:10.226+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:10.227+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:06:10.227+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:10.229+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:06:10.230+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:06:10.230+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 501 message to send. [2023-03-11T20:06:10.230+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23052, above threshold 10. [2023-03-11T20:06:11.269+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 501 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60057282 ms [2023-03-11T20:06:11.270+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:06:11.270+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:06:24.219+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch org.apache.http.NoHttpResponseException: message-router:3904 failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:06:24.220+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:06:24.220+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:06:24.220+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:06:24.227+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:06:24.227+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:24.230+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:06:24.230+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:24.236+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:06:24.236+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:24.240+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:06:24.240+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:06:24.240+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:06:39.240+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:06:39.240+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:06:39.240+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:06:39.240+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:06:39.245+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:06:39.245+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:39.248+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:06:39.248+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:39.257+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:06:39.257+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:39.263+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:06:39.263+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:06:39.263+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:06:54.264+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:06:54.264+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:06:54.264+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:06:54.264+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:06:54.271+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:06:54.271+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:54.286+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:06:54.286+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:54.306+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:06:54.306+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:06:54.309+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:06:54.309+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:06:54.309+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:07:09.309+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:07:09.310+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:07:09.310+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:07:09.310+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:07:09.317+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:07:09.317+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:09.320+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:07:09.320+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:09.322+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router:3904: The target server failed to respond [2023-03-11T20:07:09.322+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:09.326+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:07:09.326+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:07:09.326+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:07:11.050+00:00|INFO|network|pool-3-thread-1] [OUT|DMAAP|POLICY-ACRUNTIME-PARTICIPANT] {"state":"UNKNOWN","healthStatus":"UNKNOWN","participantStatistics":{"participantId":{"name":"K8sParticipant0","version":"1.0.0"},"timeStamp":"2023-03-11T20:07:11.049552Z","state":"UNKNOWN","healthStatus":"UNKNOWN","eventCount":0,"lastExecutionTime":0,"averageExecutionTime":0.0,"upTime":0,"lastEnterTime":0,"lastStart":0},"participantDefinitionUpdates":[],"automationCompositionInfoList":[],"messageType":"PARTICIPANT_STATUS","messageId":"97330b65-665a-4eb8-86c1-499ccc69a520","timestamp":"2023-03-11T20:07:11.049625Z","participantType":{"name":"org.onap.policy.clamp.acm.KubernetesParticipant","version":"2.3.4"},"participantId":{"name":"K8sParticipant0","version":"1.0.0"}} [2023-03-11T20:07:24.326+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:07:24.327+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:07:24.327+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:07:24.327+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:07:24.336+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:07:24.336+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:24.342+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:07:24.342+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:24.345+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:07:24.345+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:24.351+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:07:24.352+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:07:24.352+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:07:28.767+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Read timed out; blacklisting for 2 minutes [2023-03-11T20:07:28.767+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Read timed out java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:07:28.767+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 501 message to send. [2023-03-11T20:07:28.767+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23053, above threshold 10. [2023-03-11T20:07:29.769+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60135782 ms [2023-03-11T20:07:29.770+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:07:29.770+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:07:29.800+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:29.800+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:29.814+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:29.814+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:29.817+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:29.817+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:29.827+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Broken pipe (Write failed); blacklisting for 2 minutes [2023-03-11T20:07:29.827+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Broken pipe (Write failed) java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:07:29.827+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:07:29.827+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23054, above threshold 10. [2023-03-11T20:07:30.869+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60136882 ms [2023-03-11T20:07:30.870+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:07:30.870+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:07:30.874+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:30.874+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:30.877+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:30.877+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:30.885+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:30.885+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:30.890+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:07:30.890+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:07:30.890+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:07:30.890+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23055, above threshold 10. [2023-03-11T20:07:31.919+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60137932 ms [2023-03-11T20:07:31.920+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:07:31.920+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:07:31.924+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:31.924+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:31.935+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:31.935+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:31.944+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:31.944+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:31.965+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:07:31.965+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:07:31.965+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:07:31.965+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23056, above threshold 10. [2023-03-11T20:07:32.969+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60138982 ms [2023-03-11T20:07:32.970+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:07:32.970+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:07:32.977+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:32.977+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:32.981+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:32.981+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:32.986+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:32.986+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:32.989+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:07:32.989+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:07:32.989+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:07:32.989+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23057, above threshold 10. [2023-03-11T20:07:34.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60140032 ms [2023-03-11T20:07:34.020+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:07:34.020+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:07:34.024+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:34.024+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:34.027+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:34.027+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:34.030+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:34.030+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:34.033+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:07:34.033+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:07:34.034+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:07:34.034+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23058, above threshold 10. [2023-03-11T20:07:35.069+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60141082 ms [2023-03-11T20:07:35.070+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:07:35.070+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:07:35.072+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:35.073+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:35.074+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:35.074+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:35.077+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:35.077+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:35.080+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:07:35.081+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:07:35.081+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:07:35.081+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23059, above threshold 10. [2023-03-11T20:07:36.119+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60142132 ms [2023-03-11T20:07:36.120+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:07:36.120+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:07:36.123+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:36.123+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:36.132+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:36.132+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:36.137+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:36.137+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:36.141+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Broken pipe (Write failed); blacklisting for 2 minutes [2023-03-11T20:07:36.141+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Broken pipe (Write failed) java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:07:36.141+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:07:36.142+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23060, above threshold 10. [2023-03-11T20:07:37.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60143182 ms [2023-03-11T20:07:37.170+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:07:37.170+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:07:37.174+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:37.174+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:37.177+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:37.177+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:37.180+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:07:37.180+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:37.181+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:07:37.181+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:07:37.182+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:07:37.182+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23061, above threshold 10. [2023-03-11T20:07:38.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60144232 ms [2023-03-11T20:07:38.220+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:07:38.220+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:07:39.353+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:07:39.353+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:07:39.353+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:07:39.353+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:07:39.355+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:07:39.355+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:39.359+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:07:39.359+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:39.363+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:07:39.363+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:39.368+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:07:39.368+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:07:39.368+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:07:54.368+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:07:54.368+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:07:54.368+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:07:54.368+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:07:54.394+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:07:54.394+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:54.406+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:07:54.406+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:54.434+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:07:54.435+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:07:54.443+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:07:54.443+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:07:54.443+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:08:09.443+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:09.443+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:08:09.443+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:09.444+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:08:09.447+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:08:09.447+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:09.449+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:08:09.450+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:09.452+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:08:09.452+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:09.454+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:08:09.454+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:08:09.454+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:08:24.454+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:24.455+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:08:24.455+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:24.455+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:08:24.469+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:08:24.469+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:24.472+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:08:24.473+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:24.476+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:08:24.476+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:24.478+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. message-router:3904 failed to respond; blacklisting for 2 minutes [2023-03-11T20:08:24.478+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of message-router:3904 failed to respond [2023-03-11T20:08:24.479+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:08:38.889+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:08:38.890+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:38.900+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:38.900+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:38.906+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:38.906+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:38.914+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:38.914+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:38.914+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:38.914+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23062, above threshold 10. [2023-03-11T20:08:39.479+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch org.apache.http.NoHttpResponseException: message-router:3904 failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:39.479+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:08:39.479+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:39.479+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:08:39.482+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:08:39.482+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:39.484+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:08:39.484+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:39.487+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:08:39.487+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:39.503+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:08:39.503+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:08:39.504+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:08:39.919+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60205932 ms [2023-03-11T20:08:39.920+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:39.920+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:39.933+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:39.933+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:39.936+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:39.936+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:39.938+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:39.938+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:39.940+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:39.940+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:39.940+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:39.940+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23063, above threshold 10. [2023-03-11T20:08:40.969+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60206982 ms [2023-03-11T20:08:40.970+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:40.970+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:40.974+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:40.974+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:40.980+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:40.980+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:40.984+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:40.984+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:40.987+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:40.987+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:40.987+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:40.987+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23064, above threshold 10. [2023-03-11T20:08:42.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60208032 ms [2023-03-11T20:08:42.020+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:42.020+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:42.029+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:42.029+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:42.033+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:42.033+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:42.036+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:08:42.036+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:42.040+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:42.040+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:42.040+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:42.040+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23065, above threshold 10. [2023-03-11T20:08:43.069+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60209082 ms [2023-03-11T20:08:43.070+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:43.070+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:43.075+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:43.075+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:43.085+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:43.085+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:43.090+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:43.090+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:43.095+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:43.096+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:43.096+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:43.096+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23066, above threshold 10. [2023-03-11T20:08:44.119+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60210132 ms [2023-03-11T20:08:44.120+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:44.120+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:44.127+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:44.127+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:44.131+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:44.131+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:44.137+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:44.137+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:44.140+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:44.140+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:44.140+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:44.140+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23067, above threshold 10. [2023-03-11T20:08:45.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60211182 ms [2023-03-11T20:08:45.170+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:45.170+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:45.177+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:45.177+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:45.181+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:45.181+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:45.185+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:45.185+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:45.197+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:45.198+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:45.198+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:45.198+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23068, above threshold 10. [2023-03-11T20:08:46.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60212232 ms [2023-03-11T20:08:46.220+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:46.220+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:46.226+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:46.226+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:46.232+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:46.233+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:46.235+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:46.235+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:46.239+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:46.240+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:46.240+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:46.240+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23069, above threshold 10. [2023-03-11T20:08:47.269+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60213282 ms [2023-03-11T20:08:47.270+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:47.270+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:47.276+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:47.276+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:47.278+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:47.278+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:47.282+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:47.282+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:47.286+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:47.286+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:47.286+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:47.286+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23070, above threshold 10. [2023-03-11T20:08:48.319+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60214332 ms [2023-03-11T20:08:48.320+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:48.320+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:48.336+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:48.336+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:48.348+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:48.348+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:48.357+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:48.357+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:48.365+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:48.365+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:48.365+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:48.365+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23071, above threshold 10. [2023-03-11T20:08:49.369+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60215382 ms [2023-03-11T20:08:49.376+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:49.376+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:49.382+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:49.382+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:49.384+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:49.384+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:49.387+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:49.387+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:49.398+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:49.399+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:49.399+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:49.399+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23072, above threshold 10. [2023-03-11T20:08:50.419+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60216432 ms [2023-03-11T20:08:50.420+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:50.420+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:50.429+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:50.429+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:50.443+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:08:50.443+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:50.446+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:50.446+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:50.449+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:50.449+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:50.449+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:50.449+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23073, above threshold 10. [2023-03-11T20:08:51.469+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60217482 ms [2023-03-11T20:08:51.470+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:51.470+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:51.477+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:08:51.477+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:51.481+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:51.481+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:51.485+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:08:51.485+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:51.502+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:51.502+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:51.502+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:51.502+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23074, above threshold 10. [2023-03-11T20:08:52.519+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60218532 ms [2023-03-11T20:08:52.520+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:52.520+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:52.524+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:52.524+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:52.527+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:52.527+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:52.533+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:52.533+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:52.537+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:52.537+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:52.538+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:52.538+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23075, above threshold 10. [2023-03-11T20:08:53.569+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60219582 ms [2023-03-11T20:08:53.570+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:53.570+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:53.583+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:53.583+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:53.601+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:53.602+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:53.605+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:08:53.605+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:53.608+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:53.609+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:53.609+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:53.609+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23076, above threshold 10. [2023-03-11T20:08:54.504+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:54.504+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:08:54.504+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:54.504+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:08:54.537+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:08:54.537+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:54.579+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:08:54.579+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:54.585+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:08:54.585+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:54.603+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:08:54.603+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:08:54.603+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:08:54.619+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60220632 ms [2023-03-11T20:08:54.620+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:54.620+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:54.625+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:54.625+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:54.636+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:54.636+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:54.648+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:54.648+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:54.664+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:54.665+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:54.665+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:54.665+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23077, above threshold 10. [2023-03-11T20:08:55.669+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60221682 ms [2023-03-11T20:08:55.670+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:55.670+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:55.712+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:55.712+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:55.716+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:55.717+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:55.721+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:55.722+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:55.737+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:55.737+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:55.737+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:55.738+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23078, above threshold 10. [2023-03-11T20:08:56.769+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60222782 ms [2023-03-11T20:08:56.770+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:56.770+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:56.775+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:56.775+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:56.779+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:56.779+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:56.782+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:08:56.782+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:56.785+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:56.786+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:56.786+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:56.786+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23079, above threshold 10. [2023-03-11T20:08:57.819+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60223832 ms [2023-03-11T20:08:57.820+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:57.820+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:57.849+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:57.849+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:57.871+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:57.872+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:57.874+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:57.874+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:57.888+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:57.888+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:57.888+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:57.888+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23080, above threshold 10. [2023-03-11T20:08:58.919+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60224932 ms [2023-03-11T20:08:58.920+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:58.920+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:58.925+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:58.925+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:58.929+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:58.929+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:58.932+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:58.932+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:58.934+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:58.934+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:58.934+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:58.934+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23081, above threshold 10. [2023-03-11T20:08:59.969+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60225982 ms [2023-03-11T20:08:59.970+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:08:59.970+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:08:59.974+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:59.974+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:59.976+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:59.976+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:59.977+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:08:59.977+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:08:59.980+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:08:59.980+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:08:59.980+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:08:59.980+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23082, above threshold 10. [2023-03-11T20:09:01.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60227032 ms [2023-03-11T20:09:01.020+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:09:01.020+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:09:01.024+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:01.025+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:01.032+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:01.032+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:01.034+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:01.034+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:01.036+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:09:01.036+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:09:01.036+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:09:01.036+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23083, above threshold 10. [2023-03-11T20:09:02.069+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60228082 ms [2023-03-11T20:09:02.070+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:09:02.070+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:09:02.099+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:02.099+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:02.107+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:02.108+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:02.113+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:02.113+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:02.117+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:09:02.117+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:09:02.117+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:09:02.117+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23084, above threshold 10. [2023-03-11T20:09:03.119+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60229132 ms [2023-03-11T20:09:03.120+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:09:03.120+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:09:03.128+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:03.128+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:03.140+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:03.140+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:03.149+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:03.149+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:03.155+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:09:03.155+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:09:03.156+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:09:03.156+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23085, above threshold 10. [2023-03-11T20:09:04.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60230182 ms [2023-03-11T20:09:04.170+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:09:04.170+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:09:04.190+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:04.190+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:04.197+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:04.197+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:04.209+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:04.209+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:04.217+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:09:04.218+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:09:04.218+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:09:04.218+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23086, above threshold 10. [2023-03-11T20:09:05.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60231232 ms [2023-03-11T20:09:05.220+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:09:05.220+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:09:05.235+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:05.235+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:05.253+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:05.253+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:05.287+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:05.287+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:05.295+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:09:05.295+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:09:05.295+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:09:05.295+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23087, above threshold 10. [2023-03-11T20:09:06.319+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60232332 ms [2023-03-11T20:09:06.320+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:09:06.320+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:09:06.333+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:06.333+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:06.336+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:09:06.336+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:06.343+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:06.343+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:06.356+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:09:06.356+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:09:06.356+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:09:06.356+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23088, above threshold 10. [2023-03-11T20:09:07.369+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60233382 ms [2023-03-11T20:09:07.370+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:09:07.370+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:09:07.382+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:07.382+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:07.386+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:07.386+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:07.403+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:07.403+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:07.409+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Broken pipe (Write failed); blacklisting for 2 minutes [2023-03-11T20:09:07.409+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Broken pipe (Write failed) java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:09:07.409+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:09:07.409+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23089, above threshold 10. [2023-03-11T20:09:08.419+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60234432 ms [2023-03-11T20:09:08.420+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:09:08.420+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:09:08.445+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:08.445+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:08.477+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:08.477+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:08.507+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:09:08.507+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:09:08.513+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:09:08.513+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:09:08.513+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:09:08.513+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23090, above threshold 10. [2023-03-11T20:09:09.519+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 502 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60235532 ms [2023-03-11T20:09:09.520+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:09:09.520+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:09:09.603+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:09:09.603+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:09:09.603+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:09:09.603+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:09:11.050+00:00|INFO|network|pool-3-thread-1] [OUT|DMAAP|POLICY-ACRUNTIME-PARTICIPANT] {"state":"UNKNOWN","healthStatus":"UNKNOWN","participantStatistics":{"participantId":{"name":"K8sParticipant0","version":"1.0.0"},"timeStamp":"2023-03-11T20:09:11.049585Z","state":"UNKNOWN","healthStatus":"UNKNOWN","eventCount":0,"lastExecutionTime":0,"averageExecutionTime":0.0,"upTime":0,"lastEnterTime":0,"lastStart":0},"participantDefinitionUpdates":[],"automationCompositionInfoList":[],"messageType":"PARTICIPANT_STATUS","messageId":"ec01a2b5-b2b5-4841-b32a-c96419cd4b0d","timestamp":"2023-03-11T20:09:11.049638Z","participantType":{"name":"org.onap.policy.clamp.acm.KubernetesParticipant","version":"2.3.4"},"participantId":{"name":"K8sParticipant0","version":"1.0.0"}} [2023-03-11T20:09:54.644+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Read timed out; blacklisting for 2 minutes [2023-03-11T20:09:54.644+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Read timed out [2023-03-11T20:09:54.644+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:10:09.644+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:09.644+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:10:09.644+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:09.645+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:10:09.652+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:10:09.652+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:09.657+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:10:09.657+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:09.661+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:10:09.661+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:09.664+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:10:09.664+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:10:09.664+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:10:24.664+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:24.664+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:10:24.664+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:24.664+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:10:24.668+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router:3904: The target server failed to respond [2023-03-11T20:10:24.668+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:24.671+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:10:24.671+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:24.674+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:10:24.674+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:24.679+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:10:24.679+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:10:24.679+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:10:38.965+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:10:38.965+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:38.968+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:38.968+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:38.971+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:38.971+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:38.983+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:38.983+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:38.983+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 502 message to send. [2023-03-11T20:10:38.983+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23091, above threshold 10. [2023-03-11T20:10:39.683+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:39.683+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:10:39.683+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:39.683+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:10:39.690+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:10:39.690+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:39.695+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:10:39.695+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:39.702+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:10:39.702+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:39.711+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:10:39.711+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:10:39.711+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:10:40.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60326032 ms [2023-03-11T20:10:40.020+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:40.020+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:40.023+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:40.023+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:40.026+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:40.026+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:40.028+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:40.028+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:40.040+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:40.040+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:40.040+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:40.040+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23092, above threshold 10. [2023-03-11T20:10:41.069+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60327082 ms [2023-03-11T20:10:41.070+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:41.070+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:41.082+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:41.082+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:41.087+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:41.087+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:41.090+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:41.090+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:41.100+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:41.101+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:41.101+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:41.101+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23093, above threshold 10. [2023-03-11T20:10:42.119+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60328132 ms [2023-03-11T20:10:42.120+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:42.120+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:42.127+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:42.127+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:42.139+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:42.139+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:42.145+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:42.145+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:42.157+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:42.157+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:42.158+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:42.158+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23094, above threshold 10. [2023-03-11T20:10:43.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60329182 ms [2023-03-11T20:10:43.170+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:43.170+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:43.198+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:43.198+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:43.216+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:43.217+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:43.222+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:43.222+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:43.259+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:43.259+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:43.259+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:43.259+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23095, above threshold 10. [2023-03-11T20:10:44.269+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60330282 ms [2023-03-11T20:10:44.270+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:44.270+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:44.279+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:44.279+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:44.294+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:44.294+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:44.299+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:44.299+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:44.305+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:44.305+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:44.306+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:44.306+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23096, above threshold 10. [2023-03-11T20:10:45.319+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60331332 ms [2023-03-11T20:10:45.320+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:45.320+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:45.325+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:45.325+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:45.328+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:45.328+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:45.332+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:45.332+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:45.335+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Broken pipe (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:45.336+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Broken pipe (Write failed) java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:45.336+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:45.336+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23097, above threshold 10. [2023-03-11T20:10:46.369+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60332382 ms [2023-03-11T20:10:46.370+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:46.370+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:46.377+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:46.377+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:46.392+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:10:46.392+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:46.398+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:46.398+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:46.405+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:46.406+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:46.406+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:46.406+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23098, above threshold 10. [2023-03-11T20:10:47.419+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60333432 ms [2023-03-11T20:10:47.421+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:47.421+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:47.426+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:47.426+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:47.437+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:47.437+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:47.441+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:47.442+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:47.447+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:47.447+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:47.447+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:47.447+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23099, above threshold 10. [2023-03-11T20:10:48.469+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60334482 ms [2023-03-11T20:10:48.470+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:48.470+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:48.473+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:48.473+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:48.482+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:48.482+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:48.487+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:48.487+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:48.492+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:48.492+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:48.492+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:48.492+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23100, above threshold 10. [2023-03-11T20:10:49.519+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60335532 ms [2023-03-11T20:10:49.520+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:49.520+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:49.529+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:49.530+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:49.543+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:49.543+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:49.550+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:49.550+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:49.558+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:49.558+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:49.558+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:49.558+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23101, above threshold 10. [2023-03-11T20:10:50.569+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60336582 ms [2023-03-11T20:10:50.570+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:50.570+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:50.582+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:50.582+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:50.618+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:50.618+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:50.626+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:50.626+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:50.639+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:50.639+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:50.639+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:50.640+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23102, above threshold 10. [2023-03-11T20:10:51.669+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60337682 ms [2023-03-11T20:10:51.670+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:51.670+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:51.678+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:51.679+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:51.685+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Broken pipe (Write failed) [2023-03-11T20:10:51.685+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:51.689+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:51.689+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:51.702+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:51.702+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:51.702+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:51.702+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23103, above threshold 10. [2023-03-11T20:10:52.719+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60338732 ms [2023-03-11T20:10:52.720+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:52.720+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:52.727+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:52.727+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:52.731+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:52.731+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:52.735+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:52.735+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:52.741+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:52.742+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:52.742+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:52.742+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23104, above threshold 10. [2023-03-11T20:10:53.769+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60339782 ms [2023-03-11T20:10:53.770+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:53.770+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:53.783+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:53.783+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:53.792+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:53.792+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:53.800+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:53.800+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:53.809+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:53.809+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:53.809+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:53.809+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23105, above threshold 10. [2023-03-11T20:10:54.711+00:00|ERROR|SingleThreadedBusTopicSource|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=c085bc57-7f7c-4245-90d8-72c1ddf2c0b4, consumerInstance=onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9, fetchTimeout=15000, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=false, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-ACRUNTIME-PARTICIPANT, effectiveTopic=POLICY-ACRUNTIME-PARTICIPANT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:212) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.fetchAllMessages(SingleThreadedBusTopicSource.java:244) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:234) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:54.711+00:00|INFO|CambriaConsumerImpl|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] UEB GET /events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 [2023-03-11T20:10:54.711+00:00|WARN|HostSelector|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:54.711+00:00|INFO|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] GET http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT/c085bc57-7f7c-4245-90d8-72c1ddf2c0b4/onap-policy-clamp-ac-k8s-ppnt-547c4d78d5-n7xr9?timeout=15000 (anonymous) ... [2023-03-11T20:10:54.714+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:10:54.714+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:54.717+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:10:54.717+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:54.721+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset [2023-03-11T20:10:54.721+00:00|INFO|RetryExec|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:54.740+00:00|WARN|HttpClient|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] Error executing HTTP request. Connection reset; blacklisting for 2 minutes [2023-03-11T20:10:54.740+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Connection reset [2023-03-11T20:10:54.740+00:00|INFO|BusConsumer$FetchingBusConsumer|DMAAP-source-POLICY-ACRUNTIME-PARTICIPANT] CambriaConsumerWrapper [fetchTimeout=15000]: backoff for 15000ms [2023-03-11T20:10:54.819+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60340832 ms [2023-03-11T20:10:54.820+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:54.820+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:54.833+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:54.833+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:54.839+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:54.839+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:54.847+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:54.847+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:54.853+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:54.853+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:54.853+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:54.853+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23106, above threshold 10. [2023-03-11T20:10:55.869+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60341882 ms [2023-03-11T20:10:55.870+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:55.870+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:55.914+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:55.914+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:55.978+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:55.978+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:56.005+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:56.006+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:56.013+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:56.014+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:56.014+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:56.014+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23107, above threshold 10. [2023-03-11T20:10:57.019+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60343032 ms [2023-03-11T20:10:57.020+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:57.020+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:57.027+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:57.027+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:57.064+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:57.064+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:57.074+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:57.074+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:57.089+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:57.089+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:57.089+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:57.089+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23108, above threshold 10. [2023-03-11T20:10:58.119+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60344132 ms [2023-03-11T20:10:58.120+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:58.120+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:58.124+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:58.124+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:58.127+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:58.127+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:58.138+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:58.138+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:58.143+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:58.143+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:58.143+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:58.144+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23109, above threshold 10. [2023-03-11T20:10:59.169+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60345182 ms [2023-03-11T20:10:59.170+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:10:59.170+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:10:59.175+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:59.175+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:59.178+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:59.178+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:59.186+00:00|INFO|RetryExec|pool-4-thread-1] I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router:3904: Connection reset by peer (Write failed) [2023-03-11T20:10:59.186+00:00|INFO|RetryExec|pool-4-thread-1] Retrying request to {}->http://message-router:3904 [2023-03-11T20:10:59.200+00:00|WARN|HttpClient|pool-4-thread-1] Error executing HTTP request. Connection reset by peer (Write failed); blacklisting for 2 minutes [2023-03-11T20:10:59.200+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Connection reset by peer (Write failed) java.net.SocketException: Connection reset by peer (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112) at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:456) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:342) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:251) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:31) at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:411) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) [2023-03-11T20:10:59.200+00:00|WARN|CambriaSimplerBatchPublisher|pool-4-thread-1] Send failed, 503 message to send. [2023-03-11T20:10:59.200+00:00|ERROR|CambriaSimplerBatchPublisher|pool-4-thread-1] PUB_CHRONIC_FAILURE: Send failure count is 23110, above threshold 10. [2023-03-11T20:11:00.219+00:00|INFO|CambriaSimplerBatchPublisher|pool-4-thread-1] sending 503 msgs to /events/POLICY-ACRUNTIME-PARTICIPANT. Oldest: 60346232 ms [2023-03-11T20:11:00.220+00:00|WARN|HostSelector|pool-4-thread-1] All hosts were blacklisted; reverting to full set of hosts. [2023-03-11T20:11:00.220+00:00|INFO|HttpClient|pool-4-thread-1] POST http://message-router:3904/events/POLICY-ACRUNTIME-PARTICIPANT (anonymous) ... [2023-03-11T20:11:00.227+00:00|INFO|RetryExec|pool-4-thread-1] I/O exce