By type
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"elapsed": 0,
"match_groupdict": {},
"match_groups": [],
"path": null,
"port": 5432,
"search_regex": null,
"state": "started"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"db": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
}
Default organization added.
Demo Credential, Inventory, and Job Template added.
(changed: True)
2022-08-28 02:41:46,165 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(6bce041f-306e-4fd8-afac-b227c07c3e62, queue=awx_private_queue)
Successfully registered instance onap-awx-0
(changed: True)
2022-08-28 02:41:50,254 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(39522dc2-0bb9-4cf6-bb7e-91646b6c1df9, queue=awx_private_queue)
2022-08-28 02:41:50,316 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(f7224abe-66aa-4413-aaae-801184075940, queue=awx_private_queue)
Creating instance group tower
(changed: True)
/usr/bin/launch_awx_task.sh: line 27: unset: `#!/bin/sh': not a valid identifier
2022-08-28 02:41:51,100 INFO RPC interface 'supervisor' initialized
2022-08-28 02:41:51,100 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-08-28 02:41:51,101 INFO supervisord started with pid 150
2022-08-28 02:41:52,105 INFO spawned: 'awx-config-watcher' with pid 153
2022-08-28 02:41:52,109 INFO spawned: 'dispatcher' with pid 154
2022-08-28 02:41:52,112 INFO spawned: 'callback-receiver' with pid 155
2022-08-28 02:41:52,116 INFO spawned: 'channels-worker' with pid 156
READY
2022-08-28 02:41:53,248 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-08-28 02:41:53,249 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-08-28 02:41:53,249 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-08-28 02:41:53,249 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-08-28 02:41:55,933 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2022-08-28 02:41:55,933 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2022-08-28 02:41:55,940 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2022-08-28 02:41:55,940 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2022-08-28 02:41:55,948 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2022-08-28 02:41:55,948 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2022-08-28 02:41:55,954 - INFO - runworker - Using single-threaded worker.
2022-08-28 02:41:55,954 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer)
2022-08-28 02:41:55,955 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive
2022-08-28 02:41:55,959 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2022-08-28 02:41:55,959 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2022-08-28 02:41:55,969 WARNING awx.main.dispatch scaling up worker pid:180
2022-08-28 02:41:55,986 ERROR celery.beat Removing corrupted schedule file '/var/lib/awx/beat.db': error(11, 'Resource temporarily unavailable')
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 485, in setup_schedule
self._store = self._open_schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 475, in _open_schedule
return self.persistence.open(self.schedule_filename, writeback=True)
File "/usr/lib64/python3.6/shelve.py", line 243, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/usr/lib64/python3.6/shelve.py", line 227, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
File "/usr/lib64/python3.6/dbm/__init__.py", line 94, in open
return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable
2022-08-28 02:41:55,985 WARNING awx.main.dispatch scaling up worker pid:181
2022-08-28 02:41:56,001 WARNING awx.main.dispatch scaling up worker pid:182
2022-08-28 02:41:56,013 WARNING awx.main.dispatch scaling up worker pid:183
2022-08-28 02:41:56,022 DEBUG awx.main.tasks Syncing Schedules
2022-08-28 02:41:56,294 DEBUG awx.main.tasks Waited 0.0015878677368164062 seconds to obtain lock name: cluster_policy_lock
2022-08-28 02:41:56,310 DEBUG awx.main.tasks Total non-isolated instances:1 available for policy: 1
2022-08-28 02:41:56,310 DEBUG awx.main.tasks Policy percentage, adding Instances [1] to Group tower
2022-08-28 02:41:56,310 DEBUG awx.main.tasks Adding instances [1] to group tower
2022-08-28 02:41:56,323 DEBUG awx.main.tasks Cluster policy computation finished in 0.028131484985351562 seconds
2022-08-28 02:41:56,324 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:41:56,372 DEBUG awx.main.dispatch listening on ['awx_private_queue [direct]', 'onap-awx-0 [direct]', 'tower_broadcast_all_onap-awx-0 [fanout]']
RESULT 2
OKREADY
2022-08-28 02:42:16,276 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ab3460b9-4b8e-4d71-a9ee-cda731651a9e, queue=awx_private_queue)
[2022-08-28 02:42:16,276: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ab3460b9-4b8e-4d71-a9ee-cda731651a9e, queue=awx_private_queue)
2022-08-28 02:42:16,296 DEBUG awx.main.dispatch delivered ab3460b9-4b8e-4d71-a9ee-cda731651a9e to worker[182] qsize 0
2022-08-28 02:42:16,302 DEBUG awx.main.dispatch task ab3460b9-4b8e-4d71-a9ee-cda731651a9e starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:42:16,305 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:42:16,344 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:42:16,406 DEBUG awx.main.dispatch task ab3460b9-4b8e-4d71-a9ee-cda731651a9e is finished
2022-08-28 02:42:26,244 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(34f7b573-9043-4034-b80a-0be69e6af38a, queue=awx_private_queue)
[2022-08-28 02:42:26,244: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(34f7b573-9043-4034-b80a-0be69e6af38a, queue=awx_private_queue)
2022-08-28 02:42:26,261 DEBUG awx.main.dispatch delivered 34f7b573-9043-4034-b80a-0be69e6af38a to worker[182] qsize 0
2022-08-28 02:42:26,264 DEBUG awx.main.dispatch task 34f7b573-9043-4034-b80a-0be69e6af38a starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:42:26,280 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:42:26,292 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:42:26.288038+00:00
2022-08-28 02:42:26,314 DEBUG awx.main.dispatch task 34f7b573-9043-4034-b80a-0be69e6af38a is finished
2022-08-28 02:42:36,257 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c2812877-5bda-44f3-96c4-e4ab499f02cc, queue=awx_private_queue)
[2022-08-28 02:42:36,257: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c2812877-5bda-44f3-96c4-e4ab499f02cc, queue=awx_private_queue)
2022-08-28 02:42:36,304 DEBUG awx.main.dispatch delivered c2812877-5bda-44f3-96c4-e4ab499f02cc to worker[180] qsize 0
2022-08-28 02:42:36,309 DEBUG awx.main.dispatch task c2812877-5bda-44f3-96c4-e4ab499f02cc starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:42:36,312 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:42:36,359 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:42:36,418 DEBUG awx.main.dispatch task c2812877-5bda-44f3-96c4-e4ab499f02cc is finished
2022-08-28 02:42:56,245 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(691323f8-3413-4ae3-bb26-46ecc4a5ed2c, queue=onap-awx-0)
[2022-08-28 02:42:56,245: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(691323f8-3413-4ae3-bb26-46ecc4a5ed2c, queue=onap-awx-0)
2022-08-28 02:42:56,482 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b694135d-d216-41b4-a838-0fced17d9e8d, queue=onap-awx-0)
[2022-08-28 02:42:56,482: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b694135d-d216-41b4-a838-0fced17d9e8d, queue=onap-awx-0)
2022-08-28 02:42:56,502 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ffb154c9-50e6-41f7-8a50-92a63fcce7aa, queue=awx_private_queue)
[2022-08-28 02:42:56,502: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ffb154c9-50e6-41f7-8a50-92a63fcce7aa, queue=awx_private_queue)
2022-08-28 02:42:56,515 DEBUG awx.main.dispatch delivered 691323f8-3413-4ae3-bb26-46ecc4a5ed2c to worker[180] qsize 0
2022-08-28 02:42:56,518 DEBUG awx.main.dispatch delivered b694135d-d216-41b4-a838-0fced17d9e8d to worker[181] qsize 0
2022-08-28 02:42:56,518 DEBUG awx.main.dispatch task 691323f8-3413-4ae3-bb26-46ecc4a5ed2c starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:42:56,520 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:42:56,520 DEBUG awx.main.dispatch delivered ffb154c9-50e6-41f7-8a50-92a63fcce7aa to worker[183] qsize 0
2022-08-28 02:42:56,520 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(91058ac2-1087-49c0-991c-f77590ca927f, queue=awx_private_queue)
[2022-08-28 02:42:56,520: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(91058ac2-1087-49c0-991c-f77590ca927f, queue=awx_private_queue)
2022-08-28 02:42:56,521 DEBUG awx.main.dispatch task b694135d-d216-41b4-a838-0fced17d9e8d starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:42:56,523 DEBUG awx.main.dispatch task ffb154c9-50e6-41f7-8a50-92a63fcce7aa starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:42:56,536 DEBUG awx.main.dispatch delivered 91058ac2-1087-49c0-991c-f77590ca927f to worker[182] qsize 0
2022-08-28 02:42:56,539 DEBUG awx.main.dispatch task 91058ac2-1087-49c0-991c-f77590ca927f starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:42:56,541 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:42:56,546 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:42:56,553 DEBUG awx.main.dispatch task b694135d-d216-41b4-a838-0fced17d9e8d is finished
2022-08-28 02:42:56,554 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:42:26.281107+00:00
2022-08-28 02:42:56,567 DEBUG awx.main.dispatch task 691323f8-3413-4ae3-bb26-46ecc4a5ed2c is finished
2022-08-28 02:42:56,577 DEBUG awx.main.dispatch task ffb154c9-50e6-41f7-8a50-92a63fcce7aa is finished
2022-08-28 02:42:56,579 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:42:56,644 DEBUG awx.main.dispatch task 91058ac2-1087-49c0-991c-f77590ca927f is finished
RESULT 2
OKREADY
2022-08-28 02:43:16,520 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(74314c50-eb9f-4397-90b5-e3f75d5a57a4, queue=awx_private_queue)
[2022-08-28 02:43:16,520: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(74314c50-eb9f-4397-90b5-e3f75d5a57a4, queue=awx_private_queue)
2022-08-28 02:43:16,536 DEBUG awx.main.dispatch delivered 74314c50-eb9f-4397-90b5-e3f75d5a57a4 to worker[180] qsize 0
2022-08-28 02:43:16,538 DEBUG awx.main.dispatch task 74314c50-eb9f-4397-90b5-e3f75d5a57a4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:43:16,540 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:43:16,572 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:43:16,615 DEBUG awx.main.dispatch task 74314c50-eb9f-4397-90b5-e3f75d5a57a4 is finished
2022-08-28 02:43:26,501 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7ee100ad-776a-4c76-a030-a64ca7e30d16, queue=awx_private_queue)
[2022-08-28 02:43:26,501: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7ee100ad-776a-4c76-a030-a64ca7e30d16, queue=awx_private_queue)
2022-08-28 02:43:26,517 DEBUG awx.main.dispatch delivered 7ee100ad-776a-4c76-a030-a64ca7e30d16 to worker[182] qsize 0
2022-08-28 02:43:26,520 DEBUG awx.main.dispatch task 7ee100ad-776a-4c76-a030-a64ca7e30d16 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:43:26,536 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:43:26,540 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:42:56.546897+00:00
2022-08-28 02:43:26,567 DEBUG awx.main.dispatch task 7ee100ad-776a-4c76-a030-a64ca7e30d16 is finished
2022-08-28 02:43:36,521 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6d14c572-6b39-4fa0-b847-8c36d1b4f4ff, queue=awx_private_queue)
[2022-08-28 02:43:36,521: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6d14c572-6b39-4fa0-b847-8c36d1b4f4ff, queue=awx_private_queue)
2022-08-28 02:43:36,536 DEBUG awx.main.dispatch delivered 6d14c572-6b39-4fa0-b847-8c36d1b4f4ff to worker[182] qsize 0
2022-08-28 02:43:36,539 DEBUG awx.main.dispatch task 6d14c572-6b39-4fa0-b847-8c36d1b4f4ff starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:43:36,540 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:43:36,565 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:43:36,607 DEBUG awx.main.dispatch task 6d14c572-6b39-4fa0-b847-8c36d1b4f4ff is finished
2022-08-28 02:43:56,254 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(28fe376f-a6e1-4570-8398-e2ffd7e77443, queue=onap-awx-0)
[2022-08-28 02:43:56,254: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(28fe376f-a6e1-4570-8398-e2ffd7e77443, queue=onap-awx-0)
2022-08-28 02:43:56,497 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(feecc0f6-d291-4803-84ce-cd7af076a859, queue=onap-awx-0)
[2022-08-28 02:43:56,497: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(feecc0f6-d291-4803-84ce-cd7af076a859, queue=onap-awx-0)
2022-08-28 02:43:56,513 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ab9ac3b0-dc20-4531-b10a-d3055ca1bcaa, queue=awx_private_queue)
[2022-08-28 02:43:56,513: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ab9ac3b0-dc20-4531-b10a-d3055ca1bcaa, queue=awx_private_queue)
2022-08-28 02:43:56,516 DEBUG awx.main.dispatch delivered 28fe376f-a6e1-4570-8398-e2ffd7e77443 to worker[183] qsize 0
2022-08-28 02:43:56,519 DEBUG awx.main.dispatch delivered feecc0f6-d291-4803-84ce-cd7af076a859 to worker[182] qsize 0
2022-08-28 02:43:56,519 DEBUG awx.main.dispatch task 28fe376f-a6e1-4570-8398-e2ffd7e77443 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:43:56,520 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:43:56,521 DEBUG awx.main.dispatch task feecc0f6-d291-4803-84ce-cd7af076a859 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:43:56,530 DEBUG awx.main.dispatch delivered ab9ac3b0-dc20-4531-b10a-d3055ca1bcaa to worker[180] qsize 0
2022-08-28 02:43:56,531 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(51e390bc-2617-4f76-9dec-23a6de931f67, queue=awx_private_queue)
[2022-08-28 02:43:56,531: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(51e390bc-2617-4f76-9dec-23a6de931f67, queue=awx_private_queue)
2022-08-28 02:43:56,533 DEBUG awx.main.dispatch task ab9ac3b0-dc20-4531-b10a-d3055ca1bcaa starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:43:56,547 DEBUG awx.main.dispatch delivered 51e390bc-2617-4f76-9dec-23a6de931f67 to worker[181] qsize 0
2022-08-28 02:43:56,550 DEBUG awx.main.dispatch task 51e390bc-2617-4f76-9dec-23a6de931f67 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:43:56,552 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:43:56,603 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:43:56,628 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:43:26.536891+00:00
2022-08-28 02:43:56,634 DEBUG awx.main.dispatch task feecc0f6-d291-4803-84ce-cd7af076a859 is finished
2022-08-28 02:43:56,643 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:43:56,680 DEBUG awx.main.dispatch task 28fe376f-a6e1-4570-8398-e2ffd7e77443 is finished
2022-08-28 02:43:56,713 DEBUG awx.main.dispatch task ab9ac3b0-dc20-4531-b10a-d3055ca1bcaa is finished
2022-08-28 02:43:56,828 DEBUG awx.main.dispatch task 51e390bc-2617-4f76-9dec-23a6de931f67 is finished
RESULT 2
OKREADY
2022-08-28 02:44:16,542 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3e0a219d-e9e2-4e7a-be9b-e168fc1f94ab, queue=awx_private_queue)
[2022-08-28 02:44:16,542: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3e0a219d-e9e2-4e7a-be9b-e168fc1f94ab, queue=awx_private_queue)
2022-08-28 02:44:16,560 DEBUG awx.main.dispatch delivered 3e0a219d-e9e2-4e7a-be9b-e168fc1f94ab to worker[182] qsize 0
2022-08-28 02:44:16,563 DEBUG awx.main.dispatch task 3e0a219d-e9e2-4e7a-be9b-e168fc1f94ab starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:44:16,564 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:44:16,588 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:44:16,642 DEBUG awx.main.dispatch task 3e0a219d-e9e2-4e7a-be9b-e168fc1f94ab is finished
2022-08-28 02:44:26,514 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(47a74ed4-93de-43b0-8a51-1311d1c25772, queue=awx_private_queue)
[2022-08-28 02:44:26,514: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(47a74ed4-93de-43b0-8a51-1311d1c25772, queue=awx_private_queue)
2022-08-28 02:44:26,530 DEBUG awx.main.dispatch delivered 47a74ed4-93de-43b0-8a51-1311d1c25772 to worker[182] qsize 0
2022-08-28 02:44:26,532 DEBUG awx.main.dispatch task 47a74ed4-93de-43b0-8a51-1311d1c25772 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:44:26,547 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:44:26,550 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:43:56.603584+00:00
2022-08-28 02:44:26,567 DEBUG awx.main.dispatch task 47a74ed4-93de-43b0-8a51-1311d1c25772 is finished
2022-08-28 02:44:36,543 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d20c6f22-21a7-41ce-b418-88c914838fa9, queue=awx_private_queue)
[2022-08-28 02:44:36,543: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d20c6f22-21a7-41ce-b418-88c914838fa9, queue=awx_private_queue)
2022-08-28 02:44:36,561 DEBUG awx.main.dispatch delivered d20c6f22-21a7-41ce-b418-88c914838fa9 to worker[183] qsize 0
2022-08-28 02:44:36,565 DEBUG awx.main.dispatch task d20c6f22-21a7-41ce-b418-88c914838fa9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:44:36,566 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:44:36,589 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:44:36,640 DEBUG awx.main.dispatch task d20c6f22-21a7-41ce-b418-88c914838fa9 is finished
2022-08-28 02:44:56,263 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d2c8a4d3-3dee-4043-88e9-ca501c7e0eec, queue=onap-awx-0)
[2022-08-28 02:44:56,263: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d2c8a4d3-3dee-4043-88e9-ca501c7e0eec, queue=onap-awx-0)
2022-08-28 02:44:56,497 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(388e8fb8-ff54-4a4f-952f-e4c3d8517d78, queue=onap-awx-0)
[2022-08-28 02:44:56,497: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(388e8fb8-ff54-4a4f-952f-e4c3d8517d78, queue=onap-awx-0)
2022-08-28 02:44:56,513 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8996d95d-351b-45f9-9c13-5fb00ecbb4a7, queue=awx_private_queue)
[2022-08-28 02:44:56,513: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8996d95d-351b-45f9-9c13-5fb00ecbb4a7, queue=awx_private_queue)
2022-08-28 02:44:56,515 DEBUG awx.main.dispatch delivered d2c8a4d3-3dee-4043-88e9-ca501c7e0eec to worker[182] qsize 0
2022-08-28 02:44:56,517 DEBUG awx.main.dispatch task d2c8a4d3-3dee-4043-88e9-ca501c7e0eec starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:44:56,519 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:44:56,523 DEBUG awx.main.dispatch delivered 388e8fb8-ff54-4a4f-952f-e4c3d8517d78 to worker[180] qsize 0
2022-08-28 02:44:56,525 DEBUG awx.main.dispatch task 388e8fb8-ff54-4a4f-952f-e4c3d8517d78 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:44:56,527 DEBUG awx.main.dispatch delivered 8996d95d-351b-45f9-9c13-5fb00ecbb4a7 to worker[183] qsize 0
2022-08-28 02:44:56,529 DEBUG awx.main.dispatch task 8996d95d-351b-45f9-9c13-5fb00ecbb4a7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:44:56,542 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7dd4f8c5-992d-4bc4-a144-c704d5fecdbe, queue=awx_private_queue)
[2022-08-28 02:44:56,542: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7dd4f8c5-992d-4bc4-a144-c704d5fecdbe, queue=awx_private_queue)
2022-08-28 02:44:56,557 DEBUG awx.main.dispatch delivered 7dd4f8c5-992d-4bc4-a144-c704d5fecdbe to worker[181] qsize 0
2022-08-28 02:44:56,559 DEBUG awx.main.dispatch task 7dd4f8c5-992d-4bc4-a144-c704d5fecdbe starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:44:56,561 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:44:56,562 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:44:56,563 DEBUG awx.main.dispatch task 388e8fb8-ff54-4a4f-952f-e4c3d8517d78 is finished
2022-08-28 02:44:56,572 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:44:26.547216+00:00
2022-08-28 02:44:56,601 DEBUG awx.main.dispatch task 8996d95d-351b-45f9-9c13-5fb00ecbb4a7 is finished
2022-08-28 02:44:56,602 DEBUG awx.main.dispatch task d2c8a4d3-3dee-4043-88e9-ca501c7e0eec is finished
2022-08-28 02:44:56,668 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:44:56,721 DEBUG awx.main.dispatch task 7dd4f8c5-992d-4bc4-a144-c704d5fecdbe is finished
RESULT 2
OKREADY
2022-08-28 02:45:16,555 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a500db96-6a2f-4ad6-bb39-90dc11fac658, queue=awx_private_queue)
[2022-08-28 02:45:16,555: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a500db96-6a2f-4ad6-bb39-90dc11fac658, queue=awx_private_queue)
2022-08-28 02:45:16,570 DEBUG awx.main.dispatch delivered a500db96-6a2f-4ad6-bb39-90dc11fac658 to worker[180] qsize 0
2022-08-28 02:45:16,573 DEBUG awx.main.dispatch task a500db96-6a2f-4ad6-bb39-90dc11fac658 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:45:16,574 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:45:16,598 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:45:16,637 DEBUG awx.main.dispatch task a500db96-6a2f-4ad6-bb39-90dc11fac658 is finished
2022-08-28 02:45:26,514 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a24a80c9-bae4-4997-9970-a312f747192c, queue=awx_private_queue)
[2022-08-28 02:45:26,514: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a24a80c9-bae4-4997-9970-a312f747192c, queue=awx_private_queue)
2022-08-28 02:45:26,530 DEBUG awx.main.dispatch delivered a24a80c9-bae4-4997-9970-a312f747192c to worker[181] qsize 0
2022-08-28 02:45:26,532 DEBUG awx.main.dispatch task a24a80c9-bae4-4997-9970-a312f747192c starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:45:26,548 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:45:26,551 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:44:56.563004+00:00
2022-08-28 02:45:26,572 DEBUG awx.main.dispatch task a24a80c9-bae4-4997-9970-a312f747192c is finished
2022-08-28 02:45:36,556 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9d28e264-6086-4956-a8d9-8b7975efa37c, queue=awx_private_queue)
[2022-08-28 02:45:36,556: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9d28e264-6086-4956-a8d9-8b7975efa37c, queue=awx_private_queue)
2022-08-28 02:45:36,575 DEBUG awx.main.dispatch delivered 9d28e264-6086-4956-a8d9-8b7975efa37c to worker[181] qsize 0
2022-08-28 02:45:36,577 DEBUG awx.main.dispatch task 9d28e264-6086-4956-a8d9-8b7975efa37c starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:45:36,578 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:45:36,614 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:45:36,652 DEBUG awx.main.dispatch task 9d28e264-6086-4956-a8d9-8b7975efa37c is finished
2022-08-28 02:45:56,270 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(87005d6b-71cf-458c-9ebc-c31a119bd8a6, queue=onap-awx-0)
[2022-08-28 02:45:56,270: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(87005d6b-71cf-458c-9ebc-c31a119bd8a6, queue=onap-awx-0)
2022-08-28 02:45:56,505 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(1d7db852-74a2-4897-aa22-bea338270c0d, queue=onap-awx-0)
[2022-08-28 02:45:56,505: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(1d7db852-74a2-4897-aa22-bea338270c0d, queue=onap-awx-0)
2022-08-28 02:45:56,518 DEBUG awx.main.dispatch delivered 87005d6b-71cf-458c-9ebc-c31a119bd8a6 to worker[180] qsize 0
2022-08-28 02:45:56,520 DEBUG awx.main.dispatch task 87005d6b-71cf-458c-9ebc-c31a119bd8a6 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:45:56,522 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:45:56,521 DEBUG awx.main.dispatch delivered 1d7db852-74a2-4897-aa22-bea338270c0d to worker[182] qsize 0
2022-08-28 02:45:56,523 DEBUG awx.main.dispatch task 1d7db852-74a2-4897-aa22-bea338270c0d starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:45:56,522 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6041f04f-6586-4c19-87e1-c42eaf4bd08d, queue=awx_private_queue)
[2022-08-28 02:45:56,522: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6041f04f-6586-4c19-87e1-c42eaf4bd08d, queue=awx_private_queue)
2022-08-28 02:45:56,541 DEBUG awx.main.dispatch delivered 6041f04f-6586-4c19-87e1-c42eaf4bd08d to worker[181] qsize 0
2022-08-28 02:45:56,542 DEBUG awx.main.dispatch task 1d7db852-74a2-4897-aa22-bea338270c0d is finished
2022-08-28 02:45:56,545 DEBUG awx.main.dispatch task 6041f04f-6586-4c19-87e1-c42eaf4bd08d starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:45:56,555 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2969abd3-53c4-4931-b8f8-aeddb5483cdc, queue=awx_private_queue)
[2022-08-28 02:45:56,555: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2969abd3-53c4-4931-b8f8-aeddb5483cdc, queue=awx_private_queue)
2022-08-28 02:45:56,555 DEBUG awx.main.dispatch task 87005d6b-71cf-458c-9ebc-c31a119bd8a6 is finished
2022-08-28 02:45:56,561 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:45:56,564 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:45:26.548183+00:00
2022-08-28 02:45:56,570 DEBUG awx.main.dispatch delivered 2969abd3-53c4-4931-b8f8-aeddb5483cdc to worker[180] qsize 0
2022-08-28 02:45:56,572 DEBUG awx.main.dispatch task 2969abd3-53c4-4931-b8f8-aeddb5483cdc starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:45:56,574 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:45:56,586 DEBUG awx.main.dispatch task 6041f04f-6586-4c19-87e1-c42eaf4bd08d is finished
2022-08-28 02:45:56,592 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:45:56,632 DEBUG awx.main.dispatch task 2969abd3-53c4-4931-b8f8-aeddb5483cdc is finished
RESULT 2
OKREADY
2022-08-28 02:46:16,557 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(83d5b42e-a3cd-4424-83c0-b88180c068cf, queue=awx_private_queue)
[2022-08-28 02:46:16,557: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(83d5b42e-a3cd-4424-83c0-b88180c068cf, queue=awx_private_queue)
2022-08-28 02:46:16,587 DEBUG awx.main.dispatch delivered 83d5b42e-a3cd-4424-83c0-b88180c068cf to worker[181] qsize 0
2022-08-28 02:46:16,589 DEBUG awx.main.dispatch task 83d5b42e-a3cd-4424-83c0-b88180c068cf starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:46:16,590 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:46:16,612 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:46:16,649 DEBUG awx.main.dispatch task 83d5b42e-a3cd-4424-83c0-b88180c068cf is finished
2022-08-28 02:46:26,523 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6037cc6b-0705-4268-88dd-39c0983f0606, queue=awx_private_queue)
[2022-08-28 02:46:26,523: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6037cc6b-0705-4268-88dd-39c0983f0606, queue=awx_private_queue)
2022-08-28 02:46:26,540 DEBUG awx.main.dispatch delivered 6037cc6b-0705-4268-88dd-39c0983f0606 to worker[181] qsize 0
2022-08-28 02:46:26,542 DEBUG awx.main.dispatch task 6037cc6b-0705-4268-88dd-39c0983f0606 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:46:26,557 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:46:26,560 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:45:56.561786+00:00
2022-08-28 02:46:26,574 DEBUG awx.main.dispatch task 6037cc6b-0705-4268-88dd-39c0983f0606 is finished
2022-08-28 02:46:36,558 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e57002d4-4f72-4afb-9973-f1b090e3b79b, queue=awx_private_queue)
[2022-08-28 02:46:36,558: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e57002d4-4f72-4afb-9973-f1b090e3b79b, queue=awx_private_queue)
2022-08-28 02:46:36,574 DEBUG awx.main.dispatch delivered e57002d4-4f72-4afb-9973-f1b090e3b79b to worker[182] qsize 0
2022-08-28 02:46:36,576 DEBUG awx.main.dispatch task e57002d4-4f72-4afb-9973-f1b090e3b79b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:46:36,578 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:46:36,605 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:46:36,643 DEBUG awx.main.dispatch task e57002d4-4f72-4afb-9973-f1b090e3b79b is finished
2022-08-28 02:46:56,279 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(3a3ad7cb-1908-461f-9119-107cc5d64dd0, queue=onap-awx-0)
[2022-08-28 02:46:56,279: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(3a3ad7cb-1908-461f-9119-107cc5d64dd0, queue=onap-awx-0)
2022-08-28 02:46:56,505 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(4dbaac12-f8bc-4a39-9577-e1b8a749ebe5, queue=onap-awx-0)
[2022-08-28 02:46:56,505: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(4dbaac12-f8bc-4a39-9577-e1b8a749ebe5, queue=onap-awx-0)
2022-08-28 02:46:56,523 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(cae4c0e8-46ec-4c2f-80ff-d77360ee78cc, queue=awx_private_queue)
[2022-08-28 02:46:56,523: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(cae4c0e8-46ec-4c2f-80ff-d77360ee78cc, queue=awx_private_queue)
2022-08-28 02:46:56,527 DEBUG awx.main.dispatch delivered 3a3ad7cb-1908-461f-9119-107cc5d64dd0 to worker[182] qsize 0
2022-08-28 02:46:56,529 DEBUG awx.main.dispatch delivered 4dbaac12-f8bc-4a39-9577-e1b8a749ebe5 to worker[181] qsize 0
2022-08-28 02:46:56,529 DEBUG awx.main.dispatch task 3a3ad7cb-1908-461f-9119-107cc5d64dd0 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:46:56,531 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:46:56,534 DEBUG awx.main.dispatch task 4dbaac12-f8bc-4a39-9577-e1b8a749ebe5 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:46:56,539 DEBUG awx.main.dispatch delivered cae4c0e8-46ec-4c2f-80ff-d77360ee78cc to worker[180] qsize 0
2022-08-28 02:46:56,541 DEBUG awx.main.dispatch task cae4c0e8-46ec-4c2f-80ff-d77360ee78cc starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:46:56,557 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fbba2b90-eac9-4abe-bf72-adb5f6fde069, queue=awx_private_queue)
[2022-08-28 02:46:56,557: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fbba2b90-eac9-4abe-bf72-adb5f6fde069, queue=awx_private_queue)
2022-08-28 02:46:56,559 DEBUG awx.main.dispatch task 4dbaac12-f8bc-4a39-9577-e1b8a749ebe5 is finished
2022-08-28 02:46:56,568 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:46:56,572 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:46:26.557271+00:00
2022-08-28 02:46:56,573 DEBUG awx.main.dispatch delivered fbba2b90-eac9-4abe-bf72-adb5f6fde069 to worker[183] qsize 0
2022-08-28 02:46:56,576 DEBUG awx.main.dispatch task fbba2b90-eac9-4abe-bf72-adb5f6fde069 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:46:56,578 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:46:56,622 DEBUG awx.main.dispatch task cae4c0e8-46ec-4c2f-80ff-d77360ee78cc is finished
2022-08-28 02:46:56,628 DEBUG awx.main.dispatch task 3a3ad7cb-1908-461f-9119-107cc5d64dd0 is finished
2022-08-28 02:46:56,645 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:46:56,721 DEBUG awx.main.dispatch task fbba2b90-eac9-4abe-bf72-adb5f6fde069 is finished
RESULT 2
OKREADY
2022-08-28 02:47:16,558 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(05571305-3a12-46d6-97e9-5b7c88572027, queue=awx_private_queue)
[2022-08-28 02:47:16,558: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(05571305-3a12-46d6-97e9-5b7c88572027, queue=awx_private_queue)
2022-08-28 02:47:16,573 DEBUG awx.main.dispatch delivered 05571305-3a12-46d6-97e9-5b7c88572027 to worker[182] qsize 0
2022-08-28 02:47:16,575 DEBUG awx.main.dispatch task 05571305-3a12-46d6-97e9-5b7c88572027 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:47:16,577 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:47:16,622 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:47:16,663 DEBUG awx.main.dispatch task 05571305-3a12-46d6-97e9-5b7c88572027 is finished
2022-08-28 02:47:26,524 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4dd42fbb-8868-4bd4-80cc-4d0eebd79b11, queue=awx_private_queue)
[2022-08-28 02:47:26,524: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4dd42fbb-8868-4bd4-80cc-4d0eebd79b11, queue=awx_private_queue)
2022-08-28 02:47:26,539 DEBUG awx.main.dispatch delivered 4dd42fbb-8868-4bd4-80cc-4d0eebd79b11 to worker[182] qsize 0
2022-08-28 02:47:26,542 DEBUG awx.main.dispatch task 4dd42fbb-8868-4bd4-80cc-4d0eebd79b11 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:47:26,560 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:47:26,563 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:46:56.568222+00:00
2022-08-28 02:47:26,597 DEBUG awx.main.dispatch task 4dd42fbb-8868-4bd4-80cc-4d0eebd79b11 is finished
2022-08-28 02:47:36,557 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1f7de78b-ab53-4640-896e-86c7be809aab, queue=awx_private_queue)
[2022-08-28 02:47:36,557: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1f7de78b-ab53-4640-896e-86c7be809aab, queue=awx_private_queue)
2022-08-28 02:47:36,573 DEBUG awx.main.dispatch delivered 1f7de78b-ab53-4640-896e-86c7be809aab to worker[180] qsize 0
2022-08-28 02:47:36,575 DEBUG awx.main.dispatch task 1f7de78b-ab53-4640-896e-86c7be809aab starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:47:36,577 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:47:36,598 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:47:36,638 DEBUG awx.main.dispatch task 1f7de78b-ab53-4640-896e-86c7be809aab is finished
2022-08-28 02:47:56,286 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(eddf29c2-3e2e-42fa-9010-a4b3c934e4bb, queue=onap-awx-0)
[2022-08-28 02:47:56,286: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(eddf29c2-3e2e-42fa-9010-a4b3c934e4bb, queue=onap-awx-0)
2022-08-28 02:47:56,506 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8409f214-6b13-4825-b2df-e60a67883b5f, queue=onap-awx-0)
[2022-08-28 02:47:56,506: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8409f214-6b13-4825-b2df-e60a67883b5f, queue=onap-awx-0)
2022-08-28 02:47:56,516 DEBUG awx.main.dispatch delivered eddf29c2-3e2e-42fa-9010-a4b3c934e4bb to worker[183] qsize 0
2022-08-28 02:47:56,518 DEBUG awx.main.dispatch task eddf29c2-3e2e-42fa-9010-a4b3c934e4bb starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:47:56,520 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:47:56,525 DEBUG awx.main.dispatch delivered 8409f214-6b13-4825-b2df-e60a67883b5f to worker[182] qsize 0
2022-08-28 02:47:56,527 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(42c9bfff-dc51-47b5-86ff-ec11f79c1678, queue=awx_private_queue)
2022-08-28 02:47:56,527 DEBUG awx.main.dispatch task 8409f214-6b13-4825-b2df-e60a67883b5f starting awx.main.tasks.awx_k8s_reaper(*[])
[2022-08-28 02:47:56,527: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(42c9bfff-dc51-47b5-86ff-ec11f79c1678, queue=awx_private_queue)
2022-08-28 02:47:56,543 DEBUG awx.main.dispatch delivered 42c9bfff-dc51-47b5-86ff-ec11f79c1678 to worker[180] qsize 0
2022-08-28 02:47:56,547 DEBUG awx.main.dispatch task 8409f214-6b13-4825-b2df-e60a67883b5f is finished
2022-08-28 02:47:56,547 DEBUG awx.main.dispatch task 42c9bfff-dc51-47b5-86ff-ec11f79c1678 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:47:56,558 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(75c31e00-6d2c-4392-880c-d4db242b18e5, queue=awx_private_queue)
[2022-08-28 02:47:56,558: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(75c31e00-6d2c-4392-880c-d4db242b18e5, queue=awx_private_queue)
2022-08-28 02:47:56,573 DEBUG awx.main.dispatch delivered 75c31e00-6d2c-4392-880c-d4db242b18e5 to worker[181] qsize 0
2022-08-28 02:47:56,575 DEBUG awx.main.dispatch task 75c31e00-6d2c-4392-880c-d4db242b18e5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:47:56,577 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:47:56,591 DEBUG awx.main.dispatch task eddf29c2-3e2e-42fa-9010-a4b3c934e4bb is finished
2022-08-28 02:47:56,625 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:47:56,631 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:47:56,636 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:47:26.560963+00:00
2022-08-28 02:47:56,651 DEBUG awx.main.dispatch task 42c9bfff-dc51-47b5-86ff-ec11f79c1678 is finished
2022-08-28 02:47:56,697 DEBUG awx.main.dispatch task 75c31e00-6d2c-4392-880c-d4db242b18e5 is finished
RESULT 2
OKREADY
2022-08-28 02:48:16,566 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5761e79f-6357-4b00-a04d-387efeb7485c, queue=awx_private_queue)
[2022-08-28 02:48:16,566: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5761e79f-6357-4b00-a04d-387efeb7485c, queue=awx_private_queue)
2022-08-28 02:48:16,584 DEBUG awx.main.dispatch delivered 5761e79f-6357-4b00-a04d-387efeb7485c to worker[181] qsize 0
2022-08-28 02:48:16,586 DEBUG awx.main.dispatch task 5761e79f-6357-4b00-a04d-387efeb7485c starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:48:16,588 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:48:16,610 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:48:16,651 DEBUG awx.main.dispatch task 5761e79f-6357-4b00-a04d-387efeb7485c is finished
2022-08-28 02:48:26,527 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(991d66a4-9303-428f-9d6e-c26e6c196c72, queue=awx_private_queue)
[2022-08-28 02:48:26,527: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(991d66a4-9303-428f-9d6e-c26e6c196c72, queue=awx_private_queue)
2022-08-28 02:48:26,558 DEBUG awx.main.dispatch delivered 991d66a4-9303-428f-9d6e-c26e6c196c72 to worker[181] qsize 0
2022-08-28 02:48:26,560 DEBUG awx.main.dispatch task 991d66a4-9303-428f-9d6e-c26e6c196c72 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:48:26,586 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:48:26,598 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:47:56.632268+00:00
2022-08-28 02:48:26,720 DEBUG awx.main.dispatch task 991d66a4-9303-428f-9d6e-c26e6c196c72 is finished
2022-08-28 02:48:36,567 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3115a594-5ab9-4e31-9414-2e0a7e220d8e, queue=awx_private_queue)
[2022-08-28 02:48:36,567: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3115a594-5ab9-4e31-9414-2e0a7e220d8e, queue=awx_private_queue)
2022-08-28 02:48:36,598 DEBUG awx.main.dispatch delivered 3115a594-5ab9-4e31-9414-2e0a7e220d8e to worker[182] qsize 0
2022-08-28 02:48:36,600 DEBUG awx.main.dispatch task 3115a594-5ab9-4e31-9414-2e0a7e220d8e starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:48:36,602 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:48:36,680 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:48:36,803 DEBUG awx.main.dispatch task 3115a594-5ab9-4e31-9414-2e0a7e220d8e is finished
2022-08-28 02:48:56,296 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(87ae2be9-d01c-44a5-9ffd-923b8bb4fb80, queue=onap-awx-0)
[2022-08-28 02:48:56,296: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(87ae2be9-d01c-44a5-9ffd-923b8bb4fb80, queue=onap-awx-0)
2022-08-28 02:48:56,341 DEBUG awx.main.dispatch delivered 87ae2be9-d01c-44a5-9ffd-923b8bb4fb80 to worker[181] qsize 0
2022-08-28 02:48:56,351 DEBUG awx.main.dispatch task 87ae2be9-d01c-44a5-9ffd-923b8bb4fb80 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:48:56,353 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:48:56,467 DEBUG awx.main.dispatch task 87ae2be9-d01c-44a5-9ffd-923b8bb4fb80 is finished
2022-08-28 02:48:56,506 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(933a6dc5-3d68-438e-be8a-f19875a16259, queue=onap-awx-0)
[2022-08-28 02:48:56,506: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(933a6dc5-3d68-438e-be8a-f19875a16259, queue=onap-awx-0)
2022-08-28 02:48:56,528 DEBUG awx.main.dispatch delivered 933a6dc5-3d68-438e-be8a-f19875a16259 to worker[183] qsize 0
2022-08-28 02:48:56,529 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(79b219a8-5b6e-4bec-b3cf-7b22f32cfe71, queue=awx_private_queue)
[2022-08-28 02:48:56,529: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(79b219a8-5b6e-4bec-b3cf-7b22f32cfe71, queue=awx_private_queue)
2022-08-28 02:48:56,531 DEBUG awx.main.dispatch task 933a6dc5-3d68-438e-be8a-f19875a16259 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:48:56,544 DEBUG awx.main.dispatch delivered 79b219a8-5b6e-4bec-b3cf-7b22f32cfe71 to worker[180] qsize 0
2022-08-28 02:48:56,547 DEBUG awx.main.dispatch task 79b219a8-5b6e-4bec-b3cf-7b22f32cfe71 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:48:56,566 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(434e09f0-3feb-4c96-bf4e-3f4371c47e55, queue=awx_private_queue)
[2022-08-28 02:48:56,566: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(434e09f0-3feb-4c96-bf4e-3f4371c47e55, queue=awx_private_queue)
2022-08-28 02:48:56,577 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:48:56,582 DEBUG awx.main.dispatch delivered 434e09f0-3feb-4c96-bf4e-3f4371c47e55 to worker[182] qsize 0
2022-08-28 02:48:56,604 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:48:26.586510+00:00
2022-08-28 02:48:56,585 DEBUG awx.main.dispatch task 434e09f0-3feb-4c96-bf4e-3f4371c47e55 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:48:56,840 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:48:56,856 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:48:56,619 DEBUG awx.main.dispatch task 933a6dc5-3d68-438e-be8a-f19875a16259 is finished
2022-08-28 02:48:56,660 DEBUG awx.main.dispatch task 79b219a8-5b6e-4bec-b3cf-7b22f32cfe71 is finished
2022-08-28 02:48:57,000 DEBUG awx.main.dispatch task 434e09f0-3feb-4c96-bf4e-3f4371c47e55 is finished
RESULT 2
OKREADY
2022-08-28 02:49:16,567 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6078d2de-223d-4558-a011-8dfdf3c8cb26, queue=awx_private_queue)
[2022-08-28 02:49:16,567: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6078d2de-223d-4558-a011-8dfdf3c8cb26, queue=awx_private_queue)
2022-08-28 02:49:16,583 DEBUG awx.main.dispatch delivered 6078d2de-223d-4558-a011-8dfdf3c8cb26 to worker[180] qsize 0
2022-08-28 02:49:16,586 DEBUG awx.main.dispatch task 6078d2de-223d-4558-a011-8dfdf3c8cb26 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:49:16,588 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:49:16,649 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:49:16,903 DEBUG awx.main.dispatch task 6078d2de-223d-4558-a011-8dfdf3c8cb26 is finished
2022-08-28 02:49:26,529 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6ce296a2-e47b-4207-9325-6f48d035e4f0, queue=awx_private_queue)
[2022-08-28 02:49:26,529: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6ce296a2-e47b-4207-9325-6f48d035e4f0, queue=awx_private_queue)
2022-08-28 02:49:26,544 DEBUG awx.main.dispatch delivered 6ce296a2-e47b-4207-9325-6f48d035e4f0 to worker[182] qsize 0
2022-08-28 02:49:26,548 DEBUG awx.main.dispatch task 6ce296a2-e47b-4207-9325-6f48d035e4f0 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:49:26,564 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:49:26,567 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:48:56.577913+00:00
2022-08-28 02:49:26,594 DEBUG awx.main.dispatch task 6ce296a2-e47b-4207-9325-6f48d035e4f0 is finished
2022-08-28 02:49:36,567 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2ab91a0e-43fa-4560-9ba7-a258fbb2f26b, queue=awx_private_queue)
[2022-08-28 02:49:36,567: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2ab91a0e-43fa-4560-9ba7-a258fbb2f26b, queue=awx_private_queue)
2022-08-28 02:49:36,584 DEBUG awx.main.dispatch delivered 2ab91a0e-43fa-4560-9ba7-a258fbb2f26b to worker[180] qsize 0
2022-08-28 02:49:36,587 DEBUG awx.main.dispatch task 2ab91a0e-43fa-4560-9ba7-a258fbb2f26b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:49:36,589 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:49:36,680 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:49:36,828 DEBUG awx.main.dispatch task 2ab91a0e-43fa-4560-9ba7-a258fbb2f26b is finished
2022-08-28 02:49:56,302 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(75daf33c-102f-43c0-bbb7-39980b540b4d, queue=onap-awx-0)
[2022-08-28 02:49:56,302: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(75daf33c-102f-43c0-bbb7-39980b540b4d, queue=onap-awx-0)
2022-08-28 02:49:56,352 DEBUG awx.main.dispatch delivered 75daf33c-102f-43c0-bbb7-39980b540b4d to worker[183] qsize 0
2022-08-28 02:49:56,354 DEBUG awx.main.dispatch task 75daf33c-102f-43c0-bbb7-39980b540b4d starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:49:56,356 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:49:56,446 DEBUG awx.main.dispatch task 75daf33c-102f-43c0-bbb7-39980b540b4d is finished
2022-08-28 02:49:56,505 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(70a55205-d9d0-4225-a991-53910f7b76cf, queue=onap-awx-0)
[2022-08-28 02:49:56,505: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(70a55205-d9d0-4225-a991-53910f7b76cf, queue=onap-awx-0)
2022-08-28 02:49:56,521 DEBUG awx.main.dispatch delivered 70a55205-d9d0-4225-a991-53910f7b76cf to worker[183] qsize 0
2022-08-28 02:49:56,523 DEBUG awx.main.dispatch task 70a55205-d9d0-4225-a991-53910f7b76cf starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:49:56,530 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(00f49302-d2a9-4033-9eae-9013d3fab276, queue=awx_private_queue)
[2022-08-28 02:49:56,530: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(00f49302-d2a9-4033-9eae-9013d3fab276, queue=awx_private_queue)
2022-08-28 02:49:56,550 DEBUG awx.main.dispatch delivered 00f49302-d2a9-4033-9eae-9013d3fab276 to worker[180] qsize 0
2022-08-28 02:49:56,553 DEBUG awx.main.dispatch task 00f49302-d2a9-4033-9eae-9013d3fab276 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:49:56,567 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ad381df9-c120-4b21-9a1d-4ca939648908, queue=awx_private_queue)
[2022-08-28 02:49:56,567: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ad381df9-c120-4b21-9a1d-4ca939648908, queue=awx_private_queue)
2022-08-28 02:49:56,585 DEBUG awx.main.dispatch delivered ad381df9-c120-4b21-9a1d-4ca939648908 to worker[181] qsize 0
2022-08-28 02:49:56,612 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:49:56,617 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:49:26.564466+00:00
2022-08-28 02:49:56,622 DEBUG awx.main.dispatch task 70a55205-d9d0-4225-a991-53910f7b76cf is finished
2022-08-28 02:49:56,587 DEBUG awx.main.dispatch task ad381df9-c120-4b21-9a1d-4ca939648908 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:49:56,916 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:49:56,697 DEBUG awx.main.dispatch task 00f49302-d2a9-4033-9eae-9013d3fab276 is finished
2022-08-28 02:49:56,967 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:49:57,097 DEBUG awx.main.dispatch task ad381df9-c120-4b21-9a1d-4ca939648908 is finished
RESULT 2
OKREADY
2022-08-28 02:50:16,578 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(90b55a89-c0c7-4242-b67b-3e49bc133406, queue=awx_private_queue)
[2022-08-28 02:50:16,578: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(90b55a89-c0c7-4242-b67b-3e49bc133406, queue=awx_private_queue)
2022-08-28 02:50:16,593 DEBUG awx.main.dispatch delivered 90b55a89-c0c7-4242-b67b-3e49bc133406 to worker[182] qsize 0
2022-08-28 02:50:16,595 DEBUG awx.main.dispatch task 90b55a89-c0c7-4242-b67b-3e49bc133406 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:50:16,597 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:50:16,908 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:50:17,533 DEBUG awx.main.dispatch task 90b55a89-c0c7-4242-b67b-3e49bc133406 is finished
2022-08-28 02:50:26,529 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4764bc8d-915e-4215-abe4-aa0b50013878, queue=awx_private_queue)
[2022-08-28 02:50:26,529: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4764bc8d-915e-4215-abe4-aa0b50013878, queue=awx_private_queue)
2022-08-28 02:50:26,546 DEBUG awx.main.dispatch delivered 4764bc8d-915e-4215-abe4-aa0b50013878 to worker[180] qsize 0
2022-08-28 02:50:26,549 DEBUG awx.main.dispatch task 4764bc8d-915e-4215-abe4-aa0b50013878 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:50:27,072 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:50:27,088 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:49:56.613069+00:00
2022-08-28 02:50:27,488 DEBUG awx.main.dispatch task 4764bc8d-915e-4215-abe4-aa0b50013878 is finished
2022-08-28 02:50:36,579 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fe0bf0d9-29c1-4fee-9c3a-5a6c9ed128a8, queue=awx_private_queue)
[2022-08-28 02:50:36,579: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fe0bf0d9-29c1-4fee-9c3a-5a6c9ed128a8, queue=awx_private_queue)
2022-08-28 02:50:36,595 DEBUG awx.main.dispatch delivered fe0bf0d9-29c1-4fee-9c3a-5a6c9ed128a8 to worker[183] qsize 0
2022-08-28 02:50:36,598 DEBUG awx.main.dispatch task fe0bf0d9-29c1-4fee-9c3a-5a6c9ed128a8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:50:36,600 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:50:37,635 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:50:37,641 DEBUG awx.main.dispatch task fe0bf0d9-29c1-4fee-9c3a-5a6c9ed128a8 is finished
2022-08-28 02:50:56,302 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(3059e0c5-2407-4bcd-89cd-4e93791661e4, queue=onap-awx-0)
[2022-08-28 02:50:56,302: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(3059e0c5-2407-4bcd-89cd-4e93791661e4, queue=onap-awx-0)
2022-08-28 02:50:56,453 DEBUG awx.main.dispatch delivered 3059e0c5-2407-4bcd-89cd-4e93791661e4 to worker[180] qsize 0
2022-08-28 02:50:56,455 DEBUG awx.main.dispatch task 3059e0c5-2407-4bcd-89cd-4e93791661e4 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:50:57,501 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:50:56,505 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(be163cf3-f72b-482e-bd70-e95d789af454, queue=onap-awx-0)
[2022-08-28 02:50:56,505: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(be163cf3-f72b-482e-bd70-e95d789af454, queue=onap-awx-0)
2022-08-28 02:50:57,564 DEBUG awx.main.dispatch delivered be163cf3-f72b-482e-bd70-e95d789af454 to worker[181] qsize 0
2022-08-28 02:50:58,535 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:50:59,557 DEBUG awx.main.dispatch task 3059e0c5-2407-4bcd-89cd-4e93791661e4 is finished
2022-08-28 02:50:57,565 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(20834e0a-b84c-49c4-a6c9-3408c0969bfe, queue=awx_private_queue)
[2022-08-28 02:50:57,565: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(20834e0a-b84c-49c4-a6c9-3408c0969bfe, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 02:51:00,629 DEBUG awx.main.dispatch delivered 20834e0a-b84c-49c4-a6c9-3408c0969bfe to worker[180] qsize 0
2022-08-28 02:51:00,633 DEBUG awx.main.dispatch task 20834e0a-b84c-49c4-a6c9-3408c0969bfe starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:51:00,631 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b7ca6c9d-1236-4aa4-bb06-3d5cbd52968b, queue=awx_private_queue)
[2022-08-28 02:51:00,631: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b7ca6c9d-1236-4aa4-bb06-3d5cbd52968b, queue=awx_private_queue)
2022-08-28 02:51:00,653 DEBUG awx.main.dispatch delivered b7ca6c9d-1236-4aa4-bb06-3d5cbd52968b to worker[183] qsize 0
2022-08-28 02:51:00,656 DEBUG awx.main.dispatch task b7ca6c9d-1236-4aa4-bb06-3d5cbd52968b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:51:01,676 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:51:02,695 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:51:02,700 DEBUG awx.main.dispatch task b7ca6c9d-1236-4aa4-bb06-3d5cbd52968b is finished
2022-08-28 02:51:03,683 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:51:03,687 DEBUG awx.main.dispatch task 20834e0a-b84c-49c4-a6c9-3408c0969bfe is finished
2022-08-28 02:50:57,587 DEBUG awx.main.dispatch task be163cf3-f72b-482e-bd70-e95d789af454 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:51:05,833 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:51:05,837 DEBUG awx.main.dispatch task be163cf3-f72b-482e-bd70-e95d789af454 is finished
2022-08-28 02:51:20,647 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5fe505c5-f104-4f13-ab7b-aaa845a9c34d, queue=awx_private_queue)
[2022-08-28 02:51:20,647: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5fe505c5-f104-4f13-ab7b-aaa845a9c34d, queue=awx_private_queue)
2022-08-28 02:51:20,663 DEBUG awx.main.dispatch delivered 5fe505c5-f104-4f13-ab7b-aaa845a9c34d to worker[180] qsize 0
2022-08-28 02:51:20,666 DEBUG awx.main.dispatch task 5fe505c5-f104-4f13-ab7b-aaa845a9c34d starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:51:20,667 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:51:21,703 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:51:21,706 DEBUG awx.main.dispatch task 5fe505c5-f104-4f13-ab7b-aaa845a9c34d is finished
2022-08-28 02:51:27,565 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(edd6e108-0ed5-4853-b0b1-3d2f6be211de, queue=awx_private_queue)
[2022-08-28 02:51:27,565: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(edd6e108-0ed5-4853-b0b1-3d2f6be211de, queue=awx_private_queue)
2022-08-28 02:51:27,584 DEBUG awx.main.dispatch delivered edd6e108-0ed5-4853-b0b1-3d2f6be211de to worker[181] qsize 0
2022-08-28 02:51:27,803 DEBUG awx.main.dispatch task edd6e108-0ed5-4853-b0b1-3d2f6be211de starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:51:28,839 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:51:28,843 DEBUG awx.main.dispatch task edd6e108-0ed5-4853-b0b1-3d2f6be211de is finished
2022-08-28 02:51:40,648 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f1ae444b-135b-4e6c-877c-bf9185b95fe3, queue=awx_private_queue)
[2022-08-28 02:51:40,648: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f1ae444b-135b-4e6c-877c-bf9185b95fe3, queue=awx_private_queue)
2022-08-28 02:51:40,665 DEBUG awx.main.dispatch delivered f1ae444b-135b-4e6c-877c-bf9185b95fe3 to worker[180] qsize 0
2022-08-28 02:51:40,667 DEBUG awx.main.dispatch task f1ae444b-135b-4e6c-877c-bf9185b95fe3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:51:40,668 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:51:41,705 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:51:41,708 DEBUG awx.main.dispatch task f1ae444b-135b-4e6c-877c-bf9185b95fe3 is finished
2022-08-28 02:51:56,245 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(86cd57da-d3da-42bb-9df1-e09c8850f2cf, queue=onap-awx-0)
[2022-08-28 02:51:56,245: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(86cd57da-d3da-42bb-9df1-e09c8850f2cf, queue=onap-awx-0)
2022-08-28 02:51:56,261 DEBUG awx.main.dispatch delivered 86cd57da-d3da-42bb-9df1-e09c8850f2cf to worker[180] qsize 0
2022-08-28 02:51:56,265 DEBUG awx.main.dispatch task 86cd57da-d3da-42bb-9df1-e09c8850f2cf starting awx.main.tasks.awx_isolated_heartbeat(*[])
2022-08-28 02:51:56,267 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2022-08-28 02:51:56,303 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a0cfe79e-ea43-4d21-a614-7b5581cd6f3e, queue=onap-awx-0)
[2022-08-28 02:51:56,303: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a0cfe79e-ea43-4d21-a614-7b5581cd6f3e, queue=onap-awx-0)
2022-08-28 02:51:56,431 DEBUG awx.main.dispatch delivered a0cfe79e-ea43-4d21-a614-7b5581cd6f3e to worker[182] qsize 0
2022-08-28 02:51:56,434 DEBUG awx.main.dispatch task a0cfe79e-ea43-4d21-a614-7b5581cd6f3e starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:51:56,438 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:51:56,505 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a2efb3c8-f300-4c6e-81bb-284ac421c78f, queue=onap-awx-0)
[2022-08-28 02:51:56,505: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a2efb3c8-f300-4c6e-81bb-284ac421c78f, queue=onap-awx-0)
2022-08-28 02:51:56,520 DEBUG awx.main.dispatch delivered a2efb3c8-f300-4c6e-81bb-284ac421c78f to worker[181] qsize 0
2022-08-28 02:51:56,522 DEBUG awx.main.dispatch task a2efb3c8-f300-4c6e-81bb-284ac421c78f starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:51:57,284 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 469, in awx_isolated_heartbeat
with transaction.atomic():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/transaction.py", line 175, in __enter__
if not connection.get_autocommit():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 379, in get_autocommit
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:51:57,287 DEBUG awx.main.dispatch task 86cd57da-d3da-42bb-9df1-e09c8850f2cf is finished
2022-08-28 02:51:57,476 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:51:57,483 DEBUG awx.main.dispatch task a0cfe79e-ea43-4d21-a614-7b5581cd6f3e is finished
2022-08-28 02:51:57,543 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:51:57,548 DEBUG awx.main.dispatch task a2efb3c8-f300-4c6e-81bb-284ac421c78f is finished
2022-08-28 02:51:57,565 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(262262ac-62ea-4431-9a1b-83d2125244c6, queue=awx_private_queue)
[2022-08-28 02:51:57,565: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(262262ac-62ea-4431-9a1b-83d2125244c6, queue=awx_private_queue)
2022-08-28 02:51:57,581 DEBUG awx.main.dispatch delivered 262262ac-62ea-4431-9a1b-83d2125244c6 to worker[181] qsize 0
2022-08-28 02:51:57,584 DEBUG awx.main.dispatch task 262262ac-62ea-4431-9a1b-83d2125244c6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:51:58,600 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:51:58,604 DEBUG awx.main.dispatch task 262262ac-62ea-4431-9a1b-83d2125244c6 is finished
2022-08-28 02:52:00,648 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bea836f0-78a1-40d9-9b72-e39df128369e, queue=awx_private_queue)
[2022-08-28 02:52:00,648: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bea836f0-78a1-40d9-9b72-e39df128369e, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 02:52:00,668 DEBUG awx.main.dispatch delivered bea836f0-78a1-40d9-9b72-e39df128369e to worker[183] qsize 0
2022-08-28 02:52:00,671 DEBUG awx.main.dispatch task bea836f0-78a1-40d9-9b72-e39df128369e starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:52:00,673 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:52:01,703 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:52:01,705 DEBUG awx.main.dispatch task bea836f0-78a1-40d9-9b72-e39df128369e is finished
2022-08-28 02:52:20,659 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2a22351b-a055-4025-8056-698113198bc3, queue=awx_private_queue)
[2022-08-28 02:52:20,659: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2a22351b-a055-4025-8056-698113198bc3, queue=awx_private_queue)
2022-08-28 02:52:20,674 DEBUG awx.main.dispatch delivered 2a22351b-a055-4025-8056-698113198bc3 to worker[182] qsize 0
2022-08-28 02:52:20,676 DEBUG awx.main.dispatch task 2a22351b-a055-4025-8056-698113198bc3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:52:20,677 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:52:21,187 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:52:21,470 DEBUG awx.main.dispatch task 2a22351b-a055-4025-8056-698113198bc3 is finished
2022-08-28 02:52:27,565 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(16e6206f-56ef-4008-bdbb-4166e1439950, queue=awx_private_queue)
[2022-08-28 02:52:27,565: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(16e6206f-56ef-4008-bdbb-4166e1439950, queue=awx_private_queue)
2022-08-28 02:52:27,611 DEBUG awx.main.dispatch delivered 16e6206f-56ef-4008-bdbb-4166e1439950 to worker[180] qsize 0
2022-08-28 02:52:27,843 DEBUG awx.main.dispatch task 16e6206f-56ef-4008-bdbb-4166e1439950 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:52:27,876 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:52:27,893 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:50:27.072974+00:00
2022-08-28 02:52:27,961 DEBUG awx.main.dispatch task 16e6206f-56ef-4008-bdbb-4166e1439950 is finished
2022-08-28 02:52:40,659 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(46fd7763-b9bc-4064-b10e-f61db3baeb2d, queue=awx_private_queue)
[2022-08-28 02:52:40,659: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(46fd7763-b9bc-4064-b10e-f61db3baeb2d, queue=awx_private_queue)
2022-08-28 02:52:40,674 DEBUG awx.main.dispatch delivered 46fd7763-b9bc-4064-b10e-f61db3baeb2d to worker[183] qsize 0
2022-08-28 02:52:40,676 DEBUG awx.main.dispatch task 46fd7763-b9bc-4064-b10e-f61db3baeb2d starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:52:40,677 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:52:41,370 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:52:41,551 DEBUG awx.main.dispatch task 46fd7763-b9bc-4064-b10e-f61db3baeb2d is finished
2022-08-28 02:52:56,307 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(be6a9ff6-bdc7-497e-a606-a04a82518a4f, queue=onap-awx-0)
[2022-08-28 02:52:56,307: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(be6a9ff6-bdc7-497e-a606-a04a82518a4f, queue=onap-awx-0)
2022-08-28 02:52:56,413 DEBUG awx.main.dispatch delivered be6a9ff6-bdc7-497e-a606-a04a82518a4f to worker[180] qsize 0
2022-08-28 02:52:56,415 DEBUG awx.main.dispatch task be6a9ff6-bdc7-497e-a606-a04a82518a4f starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:52:56,417 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:52:56,423 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:52:56,425 DEBUG awx.main.dispatch task be6a9ff6-bdc7-497e-a606-a04a82518a4f is finished
2022-08-28 02:52:56,505 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(10d681de-2dd2-49d1-80f9-d23c3c14e1c0, queue=onap-awx-0)
[2022-08-28 02:52:56,505: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(10d681de-2dd2-49d1-80f9-d23c3c14e1c0, queue=onap-awx-0)
2022-08-28 02:52:56,521 DEBUG awx.main.dispatch delivered 10d681de-2dd2-49d1-80f9-d23c3c14e1c0 to worker[182] qsize 0
2022-08-28 02:52:56,523 DEBUG awx.main.dispatch task 10d681de-2dd2-49d1-80f9-d23c3c14e1c0 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:52:57,539 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:52:57,542 DEBUG awx.main.dispatch task 10d681de-2dd2-49d1-80f9-d23c3c14e1c0 is finished
2022-08-28 02:52:57,565 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(02b009e4-d735-4c86-a2db-9e6d5eff2139, queue=awx_private_queue)
[2022-08-28 02:52:57,565: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(02b009e4-d735-4c86-a2db-9e6d5eff2139, queue=awx_private_queue)
2022-08-28 02:52:57,580 DEBUG awx.main.dispatch delivered 02b009e4-d735-4c86-a2db-9e6d5eff2139 to worker[180] qsize 0
2022-08-28 02:52:57,582 DEBUG awx.main.dispatch task 02b009e4-d735-4c86-a2db-9e6d5eff2139 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:52:58,595 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:52:58,598 DEBUG awx.main.dispatch task 02b009e4-d735-4c86-a2db-9e6d5eff2139 is finished
2022-08-28 02:53:00,660 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ee606184-513d-4f42-9247-c5d2bc0ca7fa, queue=awx_private_queue)
[2022-08-28 02:53:00,660: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ee606184-513d-4f42-9247-c5d2bc0ca7fa, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 02:53:00,678 DEBUG awx.main.dispatch delivered ee606184-513d-4f42-9247-c5d2bc0ca7fa to worker[183] qsize 0
2022-08-28 02:53:00,680 DEBUG awx.main.dispatch task ee606184-513d-4f42-9247-c5d2bc0ca7fa starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:53:00,682 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:53:01,699 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:53:01,702 DEBUG awx.main.dispatch task ee606184-513d-4f42-9247-c5d2bc0ca7fa is finished
2022-08-28 02:53:20,669 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(55cad5fc-9fc9-4211-b87e-5dfd8c377fb6, queue=awx_private_queue)
[2022-08-28 02:53:20,669: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(55cad5fc-9fc9-4211-b87e-5dfd8c377fb6, queue=awx_private_queue)
2022-08-28 02:53:20,685 DEBUG awx.main.dispatch delivered 55cad5fc-9fc9-4211-b87e-5dfd8c377fb6 to worker[182] qsize 0
2022-08-28 02:53:20,687 DEBUG awx.main.dispatch task 55cad5fc-9fc9-4211-b87e-5dfd8c377fb6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:53:20,689 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:53:20,737 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:53:20,809 DEBUG awx.main.dispatch task 55cad5fc-9fc9-4211-b87e-5dfd8c377fb6 is finished
2022-08-28 02:53:27,565 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ff0b2123-566f-400c-8811-910bf9bc4603, queue=awx_private_queue)
[2022-08-28 02:53:27,565: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ff0b2123-566f-400c-8811-910bf9bc4603, queue=awx_private_queue)
2022-08-28 02:53:27,580 DEBUG awx.main.dispatch delivered ff0b2123-566f-400c-8811-910bf9bc4603 to worker[181] qsize 0
2022-08-28 02:53:27,581 DEBUG awx.main.dispatch task ff0b2123-566f-400c-8811-910bf9bc4603 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:53:27,629 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:53:27,637 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:52:27.876552+00:00
2022-08-28 02:53:27,672 DEBUG awx.main.dispatch task ff0b2123-566f-400c-8811-910bf9bc4603 is finished
2022-08-28 02:53:40,670 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ecb91bbe-2d41-4636-a8ca-6caabc2813ce, queue=awx_private_queue)
[2022-08-28 02:53:40,670: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ecb91bbe-2d41-4636-a8ca-6caabc2813ce, queue=awx_private_queue)
2022-08-28 02:53:40,688 DEBUG awx.main.dispatch delivered ecb91bbe-2d41-4636-a8ca-6caabc2813ce to worker[181] qsize 0
2022-08-28 02:53:40,894 DEBUG awx.main.dispatch task ecb91bbe-2d41-4636-a8ca-6caabc2813ce starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:53:40,896 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:53:41,066 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:53:41,370 DEBUG awx.main.dispatch task ecb91bbe-2d41-4636-a8ca-6caabc2813ce is finished
2022-08-28 02:53:56,312 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(548e700e-9b09-424a-b2cd-af66d2ee3cae, queue=onap-awx-0)
[2022-08-28 02:53:56,312: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(548e700e-9b09-424a-b2cd-af66d2ee3cae, queue=onap-awx-0)
2022-08-28 02:53:56,349 DEBUG awx.main.dispatch delivered 548e700e-9b09-424a-b2cd-af66d2ee3cae to worker[183] qsize 0
2022-08-28 02:53:56,352 DEBUG awx.main.dispatch task 548e700e-9b09-424a-b2cd-af66d2ee3cae starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:53:56,354 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:53:56,432 WARNING awx.main.tasks Rejoining the cluster as instance onap-awx-0.
2022-08-28 02:53:56,432 DEBUG awx.main.dispatch task 548e700e-9b09-424a-b2cd-af66d2ee3cae is finished
2022-08-28 02:53:56,505 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(26ea6846-abc3-4fa3-ba69-c946bfe800aa, queue=onap-awx-0)
[2022-08-28 02:53:56,505: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(26ea6846-abc3-4fa3-ba69-c946bfe800aa, queue=onap-awx-0)
2022-08-28 02:53:56,520 DEBUG awx.main.dispatch delivered 26ea6846-abc3-4fa3-ba69-c946bfe800aa to worker[180] qsize 0
2022-08-28 02:53:56,522 DEBUG awx.main.dispatch task 26ea6846-abc3-4fa3-ba69-c946bfe800aa starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:53:56,555 DEBUG awx.main.dispatch task 26ea6846-abc3-4fa3-ba69-c946bfe800aa is finished
2022-08-28 02:53:57,565 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(fa9ebd20-147d-4825-9b79-e4f7b30c7464, queue=awx_private_queue)
[2022-08-28 02:53:57,565: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(fa9ebd20-147d-4825-9b79-e4f7b30c7464, queue=awx_private_queue)
2022-08-28 02:53:57,580 DEBUG awx.main.dispatch delivered fa9ebd20-147d-4825-9b79-e4f7b30c7464 to worker[180] qsize 0
2022-08-28 02:53:57,582 DEBUG awx.main.dispatch task fa9ebd20-147d-4825-9b79-e4f7b30c7464 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:53:57,624 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:53:57,633 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:53:27.630138+00:00
2022-08-28 02:53:57,740 DEBUG awx.main.dispatch task fa9ebd20-147d-4825-9b79-e4f7b30c7464 is finished
2022-08-28 02:54:00,671 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c21274e8-91f0-4024-8b92-f6ee5c499fbb, queue=awx_private_queue)
[2022-08-28 02:54:00,671: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c21274e8-91f0-4024-8b92-f6ee5c499fbb, queue=awx_private_queue)
2022-08-28 02:54:00,688 DEBUG awx.main.dispatch delivered c21274e8-91f0-4024-8b92-f6ee5c499fbb to worker[180] qsize 0
RESULT 2
OKREADY
2022-08-28 02:54:00,691 DEBUG awx.main.dispatch task c21274e8-91f0-4024-8b92-f6ee5c499fbb starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:54:00,692 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:54:00,781 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:54:00,877 DEBUG awx.main.dispatch task c21274e8-91f0-4024-8b92-f6ee5c499fbb is finished
2022-08-28 02:54:20,671 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ca0ed0ac-9a9f-429b-808e-a1f750e96997, queue=awx_private_queue)
[2022-08-28 02:54:20,671: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ca0ed0ac-9a9f-429b-808e-a1f750e96997, queue=awx_private_queue)
2022-08-28 02:54:20,686 DEBUG awx.main.dispatch delivered ca0ed0ac-9a9f-429b-808e-a1f750e96997 to worker[181] qsize 0
2022-08-28 02:54:20,689 DEBUG awx.main.dispatch task ca0ed0ac-9a9f-429b-808e-a1f750e96997 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:54:20,691 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:54:20,733 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:54:20,839 DEBUG awx.main.dispatch task ca0ed0ac-9a9f-429b-808e-a1f750e96997 is finished
2022-08-28 02:54:27,566 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(694cc339-e891-4479-a82b-403e5d843f3e, queue=awx_private_queue)
[2022-08-28 02:54:27,566: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(694cc339-e891-4479-a82b-403e5d843f3e, queue=awx_private_queue)
2022-08-28 02:54:27,582 DEBUG awx.main.dispatch delivered 694cc339-e891-4479-a82b-403e5d843f3e to worker[181] qsize 0
2022-08-28 02:54:27,585 DEBUG awx.main.dispatch task 694cc339-e891-4479-a82b-403e5d843f3e starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:54:27,604 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:54:27,609 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:53:57.625002+00:00
2022-08-28 02:54:27,725 DEBUG awx.main.dispatch task 694cc339-e891-4479-a82b-403e5d843f3e is finished
2022-08-28 02:54:40,674 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4159f5c1-7a04-4753-8b6f-db758f97378c, queue=awx_private_queue)
[2022-08-28 02:54:40,674: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4159f5c1-7a04-4753-8b6f-db758f97378c, queue=awx_private_queue)
2022-08-28 02:54:40,748 DEBUG awx.main.dispatch delivered 4159f5c1-7a04-4753-8b6f-db758f97378c to worker[181] qsize 0
2022-08-28 02:54:40,974 DEBUG awx.main.dispatch task 4159f5c1-7a04-4753-8b6f-db758f97378c starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:54:40,977 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:54:41,284 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:54:41,458 DEBUG awx.main.dispatch task 4159f5c1-7a04-4753-8b6f-db758f97378c is finished
2022-08-28 02:54:56,315 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(dd6ece44-6761-438b-bb73-6193e1a0f94d, queue=onap-awx-0)
[2022-08-28 02:54:56,315: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(dd6ece44-6761-438b-bb73-6193e1a0f94d, queue=onap-awx-0)
2022-08-28 02:54:56,439 DEBUG awx.main.dispatch delivered dd6ece44-6761-438b-bb73-6193e1a0f94d to worker[182] qsize 0
2022-08-28 02:54:56,442 DEBUG awx.main.dispatch task dd6ece44-6761-438b-bb73-6193e1a0f94d starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:54:56,445 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:54:56,505 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(530038ab-b91e-4df9-babf-252a1015ceef, queue=onap-awx-0)
[2022-08-28 02:54:56,505: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(530038ab-b91e-4df9-babf-252a1015ceef, queue=onap-awx-0)
2022-08-28 02:54:56,521 DEBUG awx.main.dispatch delivered 530038ab-b91e-4df9-babf-252a1015ceef to worker[180] qsize 0
2022-08-28 02:54:56,523 DEBUG awx.main.dispatch task 530038ab-b91e-4df9-babf-252a1015ceef starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:54:56,612 DEBUG awx.main.dispatch task 530038ab-b91e-4df9-babf-252a1015ceef is finished
2022-08-28 02:54:56,670 DEBUG awx.main.dispatch task dd6ece44-6761-438b-bb73-6193e1a0f94d is finished
2022-08-28 02:54:57,565 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(12ec1f36-ea6d-4420-816f-89b04a395e8f, queue=awx_private_queue)
[2022-08-28 02:54:57,565: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(12ec1f36-ea6d-4420-816f-89b04a395e8f, queue=awx_private_queue)
2022-08-28 02:54:57,581 DEBUG awx.main.dispatch delivered 12ec1f36-ea6d-4420-816f-89b04a395e8f to worker[181] qsize 0
2022-08-28 02:54:57,584 DEBUG awx.main.dispatch task 12ec1f36-ea6d-4420-816f-89b04a395e8f starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:54:57,779 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:54:57,827 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:54:27.604481+00:00
2022-08-28 02:54:57,964 DEBUG awx.main.dispatch task 12ec1f36-ea6d-4420-816f-89b04a395e8f is finished
2022-08-28 02:55:00,674 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c6a1eb03-9629-4f31-8b6e-4d5eb07b69a7, queue=awx_private_queue)
[2022-08-28 02:55:00,674: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c6a1eb03-9629-4f31-8b6e-4d5eb07b69a7, queue=awx_private_queue)
2022-08-28 02:55:00,690 DEBUG awx.main.dispatch delivered c6a1eb03-9629-4f31-8b6e-4d5eb07b69a7 to worker[183] qsize 0
RESULT 2
OKREADY
2022-08-28 02:55:00,693 DEBUG awx.main.dispatch task c6a1eb03-9629-4f31-8b6e-4d5eb07b69a7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:55:00,696 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:55:01,306 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:55:01,764 DEBUG awx.main.dispatch task c6a1eb03-9629-4f31-8b6e-4d5eb07b69a7 is finished
2022-08-28 02:55:20,684 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8bd86f80-72f7-442a-9080-51e33161eb84, queue=awx_private_queue)
[2022-08-28 02:55:20,684: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8bd86f80-72f7-442a-9080-51e33161eb84, queue=awx_private_queue)
2022-08-28 02:55:20,700 DEBUG awx.main.dispatch delivered 8bd86f80-72f7-442a-9080-51e33161eb84 to worker[183] qsize 0
2022-08-28 02:55:20,703 DEBUG awx.main.dispatch task 8bd86f80-72f7-442a-9080-51e33161eb84 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:55:20,705 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:55:21,057 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:55:21,295 DEBUG awx.main.dispatch task 8bd86f80-72f7-442a-9080-51e33161eb84 is finished
2022-08-28 02:55:27,565 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(defa2e6e-b056-4ce2-a513-1e465b6e0ede, queue=awx_private_queue)
[2022-08-28 02:55:27,565: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(defa2e6e-b056-4ce2-a513-1e465b6e0ede, queue=awx_private_queue)
2022-08-28 02:55:27,584 DEBUG awx.main.dispatch delivered defa2e6e-b056-4ce2-a513-1e465b6e0ede to worker[183] qsize 0
2022-08-28 02:55:27,586 DEBUG awx.main.dispatch task defa2e6e-b056-4ce2-a513-1e465b6e0ede starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:55:27,707 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:55:27,727 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:54:57.779443+00:00
2022-08-28 02:55:28,007 DEBUG awx.main.dispatch task defa2e6e-b056-4ce2-a513-1e465b6e0ede is finished
2022-08-28 02:55:40,684 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(48648db8-77ee-4138-a8b9-829a0f9b306a, queue=awx_private_queue)
[2022-08-28 02:55:40,684: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(48648db8-77ee-4138-a8b9-829a0f9b306a, queue=awx_private_queue)
2022-08-28 02:55:40,702 DEBUG awx.main.dispatch delivered 48648db8-77ee-4138-a8b9-829a0f9b306a to worker[182] qsize 0
2022-08-28 02:55:40,705 DEBUG awx.main.dispatch task 48648db8-77ee-4138-a8b9-829a0f9b306a starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:55:40,706 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:55:56,319 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(dbcb99f6-8e4d-411a-9a24-0fdb90c512db, queue=onap-awx-0)
[2022-08-28 02:55:56,319: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(dbcb99f6-8e4d-411a-9a24-0fdb90c512db, queue=onap-awx-0)
2022-08-28 02:55:56,550 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8aef4455-a26a-4ece-be3c-019276aef7a2, queue=onap-awx-0)
[2022-08-28 02:55:56,550: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8aef4455-a26a-4ece-be3c-019276aef7a2, queue=onap-awx-0)
2022-08-28 02:55:56,569 DEBUG awx.main.dispatch delivered dbcb99f6-8e4d-411a-9a24-0fdb90c512db to worker[180] qsize 0
2022-08-28 02:55:56,572 DEBUG awx.main.dispatch delivered 8aef4455-a26a-4ece-be3c-019276aef7a2 to worker[181] qsize 0
2022-08-28 02:55:56,572 DEBUG awx.main.dispatch task dbcb99f6-8e4d-411a-9a24-0fdb90c512db starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:55:56,574 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:55:56,574 DEBUG awx.main.dispatch task 8aef4455-a26a-4ece-be3c-019276aef7a2 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:55:56,629 DEBUG awx.main.dispatch task 8aef4455-a26a-4ece-be3c-019276aef7a2 is finished
2022-08-28 02:55:56,672 DEBUG awx.main.dispatch task dbcb99f6-8e4d-411a-9a24-0fdb90c512db is finished
2022-08-28 02:55:57,567 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(272545b0-0669-4652-ac85-3d51eecc95d3, queue=awx_private_queue)
[2022-08-28 02:55:57,567: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(272545b0-0669-4652-ac85-3d51eecc95d3, queue=awx_private_queue)
2022-08-28 02:55:57,583 DEBUG awx.main.dispatch delivered 272545b0-0669-4652-ac85-3d51eecc95d3 to worker[183] qsize 0
2022-08-28 02:55:57,586 DEBUG awx.main.dispatch task 272545b0-0669-4652-ac85-3d51eecc95d3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:55:57,604 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:55:57,607 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:55:27.707699+00:00
2022-08-28 02:55:57,630 DEBUG awx.main.dispatch task 272545b0-0669-4652-ac85-3d51eecc95d3 is finished
2022-08-28 02:56:00,684 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(82da3987-d6ac-4f9d-b8b5-efa0c460f387, queue=awx_private_queue)
[2022-08-28 02:56:00,684: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(82da3987-d6ac-4f9d-b8b5-efa0c460f387, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 02:56:00,700 DEBUG awx.main.dispatch delivered 82da3987-d6ac-4f9d-b8b5-efa0c460f387 to worker[183] qsize 0
2022-08-28 02:56:00,702 DEBUG awx.main.dispatch task 82da3987-d6ac-4f9d-b8b5-efa0c460f387 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:56:00,704 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:56:00,758 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:56:00,805 DEBUG awx.main.dispatch task 82da3987-d6ac-4f9d-b8b5-efa0c460f387 is finished
2022-08-28 02:56:20,694 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(18c62b75-d2ef-484d-93b2-1b7dbd5e6aea, queue=awx_private_queue)
[2022-08-28 02:56:20,694: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(18c62b75-d2ef-484d-93b2-1b7dbd5e6aea, queue=awx_private_queue)
2022-08-28 02:56:20,711 DEBUG awx.main.dispatch delivered 18c62b75-d2ef-484d-93b2-1b7dbd5e6aea to worker[180] qsize 0
2022-08-28 02:56:20,713 DEBUG awx.main.dispatch task 18c62b75-d2ef-484d-93b2-1b7dbd5e6aea starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:56:20,715 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:56:20,835 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:56:20,965 DEBUG awx.main.dispatch task 18c62b75-d2ef-484d-93b2-1b7dbd5e6aea is finished
2022-08-28 02:56:27,566 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(48ea0fd4-5e70-40de-a3ca-c2d6c53a5f72, queue=awx_private_queue)
[2022-08-28 02:56:27,566: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(48ea0fd4-5e70-40de-a3ca-c2d6c53a5f72, queue=awx_private_queue)
2022-08-28 02:56:27,583 DEBUG awx.main.dispatch delivered 48ea0fd4-5e70-40de-a3ca-c2d6c53a5f72 to worker[180] qsize 0
2022-08-28 02:56:27,586 DEBUG awx.main.dispatch task 48ea0fd4-5e70-40de-a3ca-c2d6c53a5f72 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:56:27,648 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:56:27,651 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:55:57.604763+00:00
2022-08-28 02:56:27,729 DEBUG awx.main.dispatch task 48ea0fd4-5e70-40de-a3ca-c2d6c53a5f72 is finished
2022-08-28 02:56:40,694 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6b41383a-5d5c-4287-b28e-3183ea7f6e62, queue=awx_private_queue)
[2022-08-28 02:56:40,694: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6b41383a-5d5c-4287-b28e-3183ea7f6e62, queue=awx_private_queue)
2022-08-28 02:56:40,709 DEBUG awx.main.dispatch delivered 6b41383a-5d5c-4287-b28e-3183ea7f6e62 to worker[181] qsize 0
2022-08-28 02:56:40,711 DEBUG awx.main.dispatch task 6b41383a-5d5c-4287-b28e-3183ea7f6e62 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:56:40,713 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:56:40,778 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:56:40,981 DEBUG awx.main.dispatch task 6b41383a-5d5c-4287-b28e-3183ea7f6e62 is finished
2022-08-28 02:56:56,324 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(80b59727-c653-49de-9cd8-952593f33ec7, queue=onap-awx-0)
[2022-08-28 02:56:56,324: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(80b59727-c653-49de-9cd8-952593f33ec7, queue=onap-awx-0)
2022-08-28 02:56:56,591 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5194b3ce-6190-4fb3-90cf-64039bc220c7, queue=onap-awx-0)
[2022-08-28 02:56:56,591: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5194b3ce-6190-4fb3-90cf-64039bc220c7, queue=onap-awx-0)
2022-08-28 02:56:56,608 DEBUG awx.main.dispatch delivered 80b59727-c653-49de-9cd8-952593f33ec7 to worker[180] qsize 0
2022-08-28 02:56:56,611 DEBUG awx.main.dispatch delivered 5194b3ce-6190-4fb3-90cf-64039bc220c7 to worker[183] qsize 0
2022-08-28 02:56:56,611 DEBUG awx.main.dispatch task 80b59727-c653-49de-9cd8-952593f33ec7 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:56:56,613 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:56:56,613 DEBUG awx.main.dispatch task 5194b3ce-6190-4fb3-90cf-64039bc220c7 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:56:56,658 DEBUG awx.main.dispatch task 80b59727-c653-49de-9cd8-952593f33ec7 is finished
2022-08-28 02:56:56,661 DEBUG awx.main.dispatch task 5194b3ce-6190-4fb3-90cf-64039bc220c7 is finished
2022-08-28 02:56:57,566 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b2f87c2e-f1fc-4900-a6fd-4ced8f20ab75, queue=awx_private_queue)
[2022-08-28 02:56:57,566: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b2f87c2e-f1fc-4900-a6fd-4ced8f20ab75, queue=awx_private_queue)
2022-08-28 02:56:57,582 DEBUG awx.main.dispatch delivered b2f87c2e-f1fc-4900-a6fd-4ced8f20ab75 to worker[180] qsize 0
2022-08-28 02:56:57,585 DEBUG awx.main.dispatch task b2f87c2e-f1fc-4900-a6fd-4ced8f20ab75 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:56:57,616 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:56:57,621 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:56:27.648204+00:00
2022-08-28 02:56:57,643 DEBUG awx.main.dispatch task b2f87c2e-f1fc-4900-a6fd-4ced8f20ab75 is finished
2022-08-28 02:57:00,695 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(516410b4-5a25-4f9c-9a5a-aedc76f44364, queue=awx_private_queue)
[2022-08-28 02:57:00,695: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(516410b4-5a25-4f9c-9a5a-aedc76f44364, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 02:57:00,710 DEBUG awx.main.dispatch delivered 516410b4-5a25-4f9c-9a5a-aedc76f44364 to worker[183] qsize 0
2022-08-28 02:57:00,713 DEBUG awx.main.dispatch task 516410b4-5a25-4f9c-9a5a-aedc76f44364 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:57:00,714 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:57:00,756 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:57:00,826 DEBUG awx.main.dispatch task 516410b4-5a25-4f9c-9a5a-aedc76f44364 is finished
2022-08-28 02:57:20,695 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1c900b0f-556d-4bd6-9f7a-ec63b45d8efb, queue=awx_private_queue)
[2022-08-28 02:57:20,695: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1c900b0f-556d-4bd6-9f7a-ec63b45d8efb, queue=awx_private_queue)
2022-08-28 02:57:20,711 DEBUG awx.main.dispatch delivered 1c900b0f-556d-4bd6-9f7a-ec63b45d8efb to worker[181] qsize 0
2022-08-28 02:57:20,714 DEBUG awx.main.dispatch task 1c900b0f-556d-4bd6-9f7a-ec63b45d8efb starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:57:20,716 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:57:21,010 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:57:21,645 DEBUG awx.main.dispatch task 1c900b0f-556d-4bd6-9f7a-ec63b45d8efb is finished
2022-08-28 02:57:27,566 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a9b755c8-4156-44be-8d00-c409b79e1672, queue=awx_private_queue)
[2022-08-28 02:57:27,566: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a9b755c8-4156-44be-8d00-c409b79e1672, queue=awx_private_queue)
2022-08-28 02:57:27,585 DEBUG awx.main.dispatch delivered a9b755c8-4156-44be-8d00-c409b79e1672 to worker[180] qsize 0
2022-08-28 02:57:27,589 DEBUG awx.main.dispatch task a9b755c8-4156-44be-8d00-c409b79e1672 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:57:27,675 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:57:27,745 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:56:57.616838+00:00
2022-08-28 02:57:27,929 DEBUG awx.main.dispatch task a9b755c8-4156-44be-8d00-c409b79e1672 is finished
2022-08-28 02:57:40,695 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d83eaec2-9c8f-4df0-b259-467c9ca0003c, queue=awx_private_queue)
[2022-08-28 02:57:40,695: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d83eaec2-9c8f-4df0-b259-467c9ca0003c, queue=awx_private_queue)
2022-08-28 02:57:40,714 DEBUG awx.main.dispatch delivered d83eaec2-9c8f-4df0-b259-467c9ca0003c to worker[183] qsize 0
2022-08-28 02:57:40,718 DEBUG awx.main.dispatch task d83eaec2-9c8f-4df0-b259-467c9ca0003c starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:57:40,720 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:57:41,735 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:57:41,738 DEBUG awx.main.dispatch task d83eaec2-9c8f-4df0-b259-467c9ca0003c is finished
2022-08-28 02:57:51,330 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:57:51,336 DEBUG awx.main.dispatch task 48648db8-77ee-4138-a8b9-829a0f9b306a is finished
2022-08-28 02:57:56,327 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8839ec05-4b1f-46ef-8be9-b7a0b2a5ca60, queue=onap-awx-0)
[2022-08-28 02:57:56,327: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8839ec05-4b1f-46ef-8be9-b7a0b2a5ca60, queue=onap-awx-0)
2022-08-28 02:57:57,499 DEBUG awx.main.dispatch delivered 8839ec05-4b1f-46ef-8be9-b7a0b2a5ca60 to worker[180] qsize 0
2022-08-28 02:57:57,703 DEBUG awx.main.dispatch task 8839ec05-4b1f-46ef-8be9-b7a0b2a5ca60 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:57:57,706 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:57:57,372 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(44ce36a8-c799-40c6-8c2f-cb4159201476, queue=onap-awx-0)
[2022-08-28 02:57:57,372: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(44ce36a8-c799-40c6-8c2f-cb4159201476, queue=onap-awx-0)
2022-08-28 02:57:58,423 DEBUG awx.main.dispatch delivered 44ce36a8-c799-40c6-8c2f-cb4159201476 to worker[181] qsize 0
2022-08-28 02:57:58,424 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dcb4f810-f0bd-45df-a2f9-bb7d7cf3d017, queue=awx_private_queue)
[2022-08-28 02:57:58,424: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dcb4f810-f0bd-45df-a2f9-bb7d7cf3d017, queue=awx_private_queue)
2022-08-28 02:57:58,426 DEBUG awx.main.dispatch task 44ce36a8-c799-40c6-8c2f-cb4159201476 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:57:58,439 DEBUG awx.main.dispatch delivered dcb4f810-f0bd-45df-a2f9-bb7d7cf3d017 to worker[182] qsize 0
2022-08-28 02:57:58,441 DEBUG awx.main.dispatch task dcb4f810-f0bd-45df-a2f9-bb7d7cf3d017 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:57:58,723 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:57:58,726 DEBUG awx.main.dispatch task 8839ec05-4b1f-46ef-8be9-b7a0b2a5ca60 is finished
2022-08-28 02:57:59,461 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:57:59,464 DEBUG awx.main.dispatch task dcb4f810-f0bd-45df-a2f9-bb7d7cf3d017 is finished
2022-08-28 02:58:00,695 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a8a8dcea-2a73-4d7b-901c-5034f15053a4, queue=awx_private_queue)
RESULT 2
OKREADY
[2022-08-28 02:58:00,695: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a8a8dcea-2a73-4d7b-901c-5034f15053a4, queue=awx_private_queue)
2022-08-28 02:58:00,715 DEBUG awx.main.dispatch delivered a8a8dcea-2a73-4d7b-901c-5034f15053a4 to worker[182] qsize 0
2022-08-28 02:58:00,717 DEBUG awx.main.dispatch task a8a8dcea-2a73-4d7b-901c-5034f15053a4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:58:00,718 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:58:00,903 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:58:01,038 DEBUG awx.main.dispatch task a8a8dcea-2a73-4d7b-901c-5034f15053a4 is finished
2022-08-28 02:58:20,695 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6fc70dad-2043-48dc-afc1-5400c9d59d16, queue=awx_private_queue)
[2022-08-28 02:58:20,695: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6fc70dad-2043-48dc-afc1-5400c9d59d16, queue=awx_private_queue)
2022-08-28 02:58:20,713 DEBUG awx.main.dispatch delivered 6fc70dad-2043-48dc-afc1-5400c9d59d16 to worker[182] qsize 0
2022-08-28 02:58:20,716 DEBUG awx.main.dispatch task 6fc70dad-2043-48dc-afc1-5400c9d59d16 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:58:20,718 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:58:20,743 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:58:20,796 DEBUG awx.main.dispatch task 6fc70dad-2043-48dc-afc1-5400c9d59d16 is finished
2022-08-28 02:58:28,425 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c7bb9bc7-a861-4fd5-9617-8240d549856e, queue=awx_private_queue)
[2022-08-28 02:58:28,425: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c7bb9bc7-a861-4fd5-9617-8240d549856e, queue=awx_private_queue)
2022-08-28 02:58:28,441 DEBUG awx.main.dispatch delivered c7bb9bc7-a861-4fd5-9617-8240d549856e to worker[182] qsize 0
2022-08-28 02:58:28,443 DEBUG awx.main.dispatch task c7bb9bc7-a861-4fd5-9617-8240d549856e starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:58:28,489 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:58:28,501 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:57:27.675635+00:00
2022-08-28 02:58:30,787 DEBUG awx.main.dispatch task c7bb9bc7-a861-4fd5-9617-8240d549856e is finished
2022-08-28 02:58:40,698 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a29c6ae4-31b4-4d55-ad98-6a8e3d812945, queue=awx_private_queue)
[2022-08-28 02:58:40,698: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a29c6ae4-31b4-4d55-ad98-6a8e3d812945, queue=awx_private_queue)
2022-08-28 02:58:40,713 DEBUG awx.main.dispatch delivered a29c6ae4-31b4-4d55-ad98-6a8e3d812945 to worker[182] qsize 0
2022-08-28 02:58:40,715 DEBUG awx.main.dispatch task a29c6ae4-31b4-4d55-ad98-6a8e3d812945 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:58:40,716 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:58:41,906 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 02:58:42,921 DEBUG awx.main.dispatch task a29c6ae4-31b4-4d55-ad98-6a8e3d812945 is finished
2022-08-28 02:58:56,330 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(9d3f3255-f447-49e9-96a2-0d84924fc042, queue=onap-awx-0)
[2022-08-28 02:58:56,330: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(9d3f3255-f447-49e9-96a2-0d84924fc042, queue=onap-awx-0)
2022-08-28 02:58:56,405 DEBUG awx.main.dispatch delivered 9d3f3255-f447-49e9-96a2-0d84924fc042 to worker[182] qsize 0
2022-08-28 02:58:56,408 DEBUG awx.main.dispatch task 9d3f3255-f447-49e9-96a2-0d84924fc042 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:58:56,411 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:58:56,491 DEBUG awx.main.dispatch task 9d3f3255-f447-49e9-96a2-0d84924fc042 is finished
2022-08-28 02:58:57,372 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(4dc2790c-46b8-4064-a65b-f4db15578eab, queue=onap-awx-0)
[2022-08-28 02:58:57,372: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(4dc2790c-46b8-4064-a65b-f4db15578eab, queue=onap-awx-0)
2022-08-28 02:58:57,587 DEBUG awx.main.dispatch delivered 4dc2790c-46b8-4064-a65b-f4db15578eab to worker[180] qsize 0
2022-08-28 02:58:57,925 DEBUG awx.main.dispatch task 4dc2790c-46b8-4064-a65b-f4db15578eab starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:58:58,022 DEBUG awx.main.dispatch task 4dc2790c-46b8-4064-a65b-f4db15578eab is finished
2022-08-28 02:58:58,425 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ab187fea-53fa-4ba8-9f7f-06fb1aa1b0c1, queue=awx_private_queue)
[2022-08-28 02:58:58,425: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ab187fea-53fa-4ba8-9f7f-06fb1aa1b0c1, queue=awx_private_queue)
2022-08-28 02:58:58,440 DEBUG awx.main.dispatch delivered ab187fea-53fa-4ba8-9f7f-06fb1aa1b0c1 to worker[180] qsize 0
2022-08-28 02:58:58,442 DEBUG awx.main.dispatch task ab187fea-53fa-4ba8-9f7f-06fb1aa1b0c1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:58:58,505 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 02:58:58,510 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:58:28.489707+00:00
2022-08-28 02:58:58,671 DEBUG awx.main.dispatch task ab187fea-53fa-4ba8-9f7f-06fb1aa1b0c1 is finished
2022-08-28 02:59:00,698 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(39b760c7-7ea7-4e76-bd95-060a5a280359, queue=awx_private_queue)
[2022-08-28 02:59:00,698: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(39b760c7-7ea7-4e76-bd95-060a5a280359, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 02:59:00,714 DEBUG awx.main.dispatch delivered 39b760c7-7ea7-4e76-bd95-060a5a280359 to worker[182] qsize 0
2022-08-28 02:59:00,716 DEBUG awx.main.dispatch task 39b760c7-7ea7-4e76-bd95-060a5a280359 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:59:00,717 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:59:01,735 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:59:01,738 DEBUG awx.main.dispatch task 39b760c7-7ea7-4e76-bd95-060a5a280359 is finished
2022-08-28 02:59:20,708 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(10681703-e198-4487-b270-907f0a50a672, queue=awx_private_queue)
[2022-08-28 02:59:20,708: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(10681703-e198-4487-b270-907f0a50a672, queue=awx_private_queue)
2022-08-28 02:59:20,724 DEBUG awx.main.dispatch delivered 10681703-e198-4487-b270-907f0a50a672 to worker[180] qsize 0
2022-08-28 02:59:20,726 DEBUG awx.main.dispatch task 10681703-e198-4487-b270-907f0a50a672 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:59:20,728 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:59:21,764 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:59:21,767 DEBUG awx.main.dispatch task 10681703-e198-4487-b270-907f0a50a672 is finished
2022-08-28 02:59:28,425 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a8c9c4a2-5f33-4999-8717-94ff45198807, queue=awx_private_queue)
[2022-08-28 02:59:28,425: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a8c9c4a2-5f33-4999-8717-94ff45198807, queue=awx_private_queue)
2022-08-28 02:59:28,441 DEBUG awx.main.dispatch delivered a8c9c4a2-5f33-4999-8717-94ff45198807 to worker[180] qsize 0
2022-08-28 02:59:28,443 DEBUG awx.main.dispatch task a8c9c4a2-5f33-4999-8717-94ff45198807 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:59:29,480 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:59:29,483 DEBUG awx.main.dispatch task a8c9c4a2-5f33-4999-8717-94ff45198807 is finished
2022-08-28 02:59:40,712 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1aba1bdd-e1c5-402d-8b9b-78bad0db890f, queue=awx_private_queue)
[2022-08-28 02:59:40,712: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1aba1bdd-e1c5-402d-8b9b-78bad0db890f, queue=awx_private_queue)
2022-08-28 02:59:40,729 DEBUG awx.main.dispatch delivered 1aba1bdd-e1c5-402d-8b9b-78bad0db890f to worker[182] qsize 0
2022-08-28 02:59:40,731 DEBUG awx.main.dispatch task 1aba1bdd-e1c5-402d-8b9b-78bad0db890f starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 02:59:40,733 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 02:59:41,764 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:59:41,767 DEBUG awx.main.dispatch task 1aba1bdd-e1c5-402d-8b9b-78bad0db890f is finished
2022-08-28 02:59:56,335 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(fa7ea283-93a4-4b5b-ab17-296d43a3979f, queue=onap-awx-0)
[2022-08-28 02:59:56,335: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(fa7ea283-93a4-4b5b-ab17-296d43a3979f, queue=onap-awx-0)
2022-08-28 02:59:57,455 DEBUG awx.main.dispatch delivered fa7ea283-93a4-4b5b-ab17-296d43a3979f to worker[182] qsize 0
2022-08-28 02:59:57,399 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(23de2f42-64cc-4887-8388-44ee611f0ec5, queue=onap-awx-0)
[2022-08-28 02:59:57,399: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(23de2f42-64cc-4887-8388-44ee611f0ec5, queue=onap-awx-0)
2022-08-28 02:59:58,454 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0bc3a5d4-d391-47f8-a35c-9996dfd4ab96, queue=awx_private_queue)
[2022-08-28 02:59:58,454: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0bc3a5d4-d391-47f8-a35c-9996dfd4ab96, queue=awx_private_queue)
2022-08-28 02:59:57,462 DEBUG awx.main.dispatch task fa7ea283-93a4-4b5b-ab17-296d43a3979f starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 02:59:58,505 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 02:59:58,454 DEBUG awx.main.dispatch delivered 23de2f42-64cc-4887-8388-44ee611f0ec5 to worker[183] qsize 0
2022-08-28 02:59:58,666 DEBUG awx.main.dispatch delivered 0bc3a5d4-d391-47f8-a35c-9996dfd4ab96 to worker[180] qsize 0
2022-08-28 02:59:58,666 DEBUG awx.main.dispatch task 23de2f42-64cc-4887-8388-44ee611f0ec5 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 02:59:58,668 DEBUG awx.main.dispatch task 0bc3a5d4-d391-47f8-a35c-9996dfd4ab96 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 02:59:59,524 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 02:59:59,527 DEBUG awx.main.dispatch task fa7ea283-93a4-4b5b-ab17-296d43a3979f is finished
2022-08-28 03:00:00,711 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f58b3ff5-fb0e-4320-979c-3cb7bae487db, queue=awx_private_queue)
[2022-08-28 03:00:00,711: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f58b3ff5-fb0e-4320-979c-3cb7bae487db, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:00:00,726 DEBUG awx.main.dispatch delivered f58b3ff5-fb0e-4320-979c-3cb7bae487db to worker[182] qsize 0
2022-08-28 03:00:00,728 DEBUG awx.main.dispatch task f58b3ff5-fb0e-4320-979c-3cb7bae487db starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:00:00,729 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:00:01,070 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:00:01,410 DEBUG awx.main.dispatch task f58b3ff5-fb0e-4320-979c-3cb7bae487db is finished
2022-08-28 03:00:08,546 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:00:08,549 DEBUG awx.main.dispatch task 44ce36a8-c799-40c6-8c2f-cb4159201476 is finished
2022-08-28 03:00:20,711 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3bc325f8-146a-4e15-ac2d-fe7e516ec3e7, queue=awx_private_queue)
[2022-08-28 03:00:20,711: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3bc325f8-146a-4e15-ac2d-fe7e516ec3e7, queue=awx_private_queue)
2022-08-28 03:00:20,727 DEBUG awx.main.dispatch delivered 3bc325f8-146a-4e15-ac2d-fe7e516ec3e7 to worker[182] qsize 0
2022-08-28 03:00:20,729 DEBUG awx.main.dispatch task 3bc325f8-146a-4e15-ac2d-fe7e516ec3e7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:00:20,730 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:00:20,822 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:00:20,913 DEBUG awx.main.dispatch task 3bc325f8-146a-4e15-ac2d-fe7e516ec3e7 is finished
2022-08-28 03:00:28,454 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6f17e7d1-6046-42cf-beb3-184a4b33bcc5, queue=awx_private_queue)
[2022-08-28 03:00:28,454: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6f17e7d1-6046-42cf-beb3-184a4b33bcc5, queue=awx_private_queue)
2022-08-28 03:00:28,469 DEBUG awx.main.dispatch delivered 6f17e7d1-6046-42cf-beb3-184a4b33bcc5 to worker[181] qsize 0
2022-08-28 03:00:28,470 DEBUG awx.main.dispatch task 6f17e7d1-6046-42cf-beb3-184a4b33bcc5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:00:28,531 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:00:28,554 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 02:58:58.505832+00:00
2022-08-28 03:00:28,744 DEBUG awx.main.dispatch task 6f17e7d1-6046-42cf-beb3-184a4b33bcc5 is finished
2022-08-28 03:00:40,714 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(646d8b9d-4cf9-409f-b95e-0f6d4c6b81aa, queue=awx_private_queue)
[2022-08-28 03:00:40,714: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(646d8b9d-4cf9-409f-b95e-0f6d4c6b81aa, queue=awx_private_queue)
2022-08-28 03:00:40,732 DEBUG awx.main.dispatch delivered 646d8b9d-4cf9-409f-b95e-0f6d4c6b81aa to worker[182] qsize 0
2022-08-28 03:00:40,735 DEBUG awx.main.dispatch task 646d8b9d-4cf9-409f-b95e-0f6d4c6b81aa starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:00:40,738 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:00:40,803 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:00:40,896 DEBUG awx.main.dispatch task 646d8b9d-4cf9-409f-b95e-0f6d4c6b81aa is finished
2022-08-28 03:00:56,340 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(be669abd-c33b-4d11-b1d1-8d16c65f53f6, queue=onap-awx-0)
[2022-08-28 03:00:56,340: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(be669abd-c33b-4d11-b1d1-8d16c65f53f6, queue=onap-awx-0)
2022-08-28 03:00:56,377 DEBUG awx.main.dispatch delivered be669abd-c33b-4d11-b1d1-8d16c65f53f6 to worker[182] qsize 0
2022-08-28 03:00:56,379 DEBUG awx.main.dispatch task be669abd-c33b-4d11-b1d1-8d16c65f53f6 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:00:56,381 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:00:56,479 DEBUG awx.main.dispatch task be669abd-c33b-4d11-b1d1-8d16c65f53f6 is finished
2022-08-28 03:00:57,400 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(6a104950-958e-4d9e-9872-d12230239775, queue=onap-awx-0)
[2022-08-28 03:00:57,400: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(6a104950-958e-4d9e-9872-d12230239775, queue=onap-awx-0)
2022-08-28 03:00:57,414 DEBUG awx.main.dispatch delivered 6a104950-958e-4d9e-9872-d12230239775 to worker[182] qsize 0
2022-08-28 03:00:57,417 DEBUG awx.main.dispatch task 6a104950-958e-4d9e-9872-d12230239775 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:00:57,433 DEBUG awx.main.dispatch task 6a104950-958e-4d9e-9872-d12230239775 is finished
2022-08-28 03:00:58,454 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1671166f-8b9d-4500-b2eb-02b271e1f6ef, queue=awx_private_queue)
[2022-08-28 03:00:58,454: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1671166f-8b9d-4500-b2eb-02b271e1f6ef, queue=awx_private_queue)
2022-08-28 03:00:58,517 DEBUG awx.main.dispatch delivered 1671166f-8b9d-4500-b2eb-02b271e1f6ef to worker[181] qsize 0
2022-08-28 03:00:58,698 DEBUG awx.main.dispatch task 1671166f-8b9d-4500-b2eb-02b271e1f6ef starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:00:58,796 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:00:58,830 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:00:28.531738+00:00
2022-08-28 03:00:58,901 DEBUG awx.main.dispatch task 1671166f-8b9d-4500-b2eb-02b271e1f6ef is finished
2022-08-28 03:01:00,714 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(15a8b173-c47e-4961-b898-88a64429f752, queue=awx_private_queue)
[2022-08-28 03:01:00,714: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(15a8b173-c47e-4961-b898-88a64429f752, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:01:00,730 DEBUG awx.main.dispatch delivered 15a8b173-c47e-4961-b898-88a64429f752 to worker[181] qsize 0
2022-08-28 03:01:00,732 DEBUG awx.main.dispatch task 15a8b173-c47e-4961-b898-88a64429f752 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:01:00,734 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:01:00,822 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:01:00,896 DEBUG awx.main.dispatch task 15a8b173-c47e-4961-b898-88a64429f752 is finished
2022-08-28 03:01:20,715 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(66e70d85-11ff-43a2-8600-4e2f550fd5eb, queue=awx_private_queue)
[2022-08-28 03:01:20,715: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(66e70d85-11ff-43a2-8600-4e2f550fd5eb, queue=awx_private_queue)
2022-08-28 03:01:20,732 DEBUG awx.main.dispatch delivered 66e70d85-11ff-43a2-8600-4e2f550fd5eb to worker[181] qsize 0
2022-08-28 03:01:20,735 DEBUG awx.main.dispatch task 66e70d85-11ff-43a2-8600-4e2f550fd5eb starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:01:20,738 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:01:20,762 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:01:20,820 DEBUG awx.main.dispatch task 66e70d85-11ff-43a2-8600-4e2f550fd5eb is finished
2022-08-28 03:01:28,454 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6453c78a-c29f-4fe1-99c7-ab2745f290bb, queue=awx_private_queue)
[2022-08-28 03:01:28,454: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6453c78a-c29f-4fe1-99c7-ab2745f290bb, queue=awx_private_queue)
2022-08-28 03:01:28,470 DEBUG awx.main.dispatch delivered 6453c78a-c29f-4fe1-99c7-ab2745f290bb to worker[181] qsize 0
2022-08-28 03:01:28,473 DEBUG awx.main.dispatch task 6453c78a-c29f-4fe1-99c7-ab2745f290bb starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:01:28,536 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:01:28,544 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:00:58.796564+00:00
2022-08-28 03:01:28,647 DEBUG awx.main.dispatch task 6453c78a-c29f-4fe1-99c7-ab2745f290bb is finished
2022-08-28 03:01:40,719 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(11f4520f-a61f-4c7b-9e63-f4a2dac49194, queue=awx_private_queue)
[2022-08-28 03:01:40,719: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(11f4520f-a61f-4c7b-9e63-f4a2dac49194, queue=awx_private_queue)
2022-08-28 03:01:40,735 DEBUG awx.main.dispatch delivered 11f4520f-a61f-4c7b-9e63-f4a2dac49194 to worker[182] qsize 0
2022-08-28 03:01:40,737 DEBUG awx.main.dispatch task 11f4520f-a61f-4c7b-9e63-f4a2dac49194 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:01:40,739 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:01:40,825 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:01:41,021 DEBUG awx.main.dispatch task 11f4520f-a61f-4c7b-9e63-f4a2dac49194 is finished
2022-08-28 03:01:56,250 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(3c01c6cf-f1c0-445a-bd2f-ca91f2249bc2, queue=onap-awx-0)
[2022-08-28 03:01:56,250: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(3c01c6cf-f1c0-445a-bd2f-ca91f2249bc2, queue=onap-awx-0)
2022-08-28 03:01:56,266 DEBUG awx.main.dispatch delivered 3c01c6cf-f1c0-445a-bd2f-ca91f2249bc2 to worker[181] qsize 0
2022-08-28 03:01:56,269 DEBUG awx.main.dispatch task 3c01c6cf-f1c0-445a-bd2f-ca91f2249bc2 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2022-08-28 03:01:56,271 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2022-08-28 03:01:56,313 DEBUG awx.main.dispatch task 3c01c6cf-f1c0-445a-bd2f-ca91f2249bc2 is finished
2022-08-28 03:01:56,339 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(45b65744-5bb7-49a6-a1af-eb36c53fae5a, queue=onap-awx-0)
[2022-08-28 03:01:56,339: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(45b65744-5bb7-49a6-a1af-eb36c53fae5a, queue=onap-awx-0)
2022-08-28 03:01:56,382 DEBUG awx.main.dispatch delivered 45b65744-5bb7-49a6-a1af-eb36c53fae5a to worker[181] qsize 0
2022-08-28 03:01:56,385 DEBUG awx.main.dispatch task 45b65744-5bb7-49a6-a1af-eb36c53fae5a starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:01:56,386 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:01:56,419 DEBUG awx.main.dispatch task 45b65744-5bb7-49a6-a1af-eb36c53fae5a is finished
2022-08-28 03:01:57,400 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(fbb3e230-f39d-4d30-a8da-a3d21096ea99, queue=onap-awx-0)
[2022-08-28 03:01:57,400: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(fbb3e230-f39d-4d30-a8da-a3d21096ea99, queue=onap-awx-0)
2022-08-28 03:01:57,418 DEBUG awx.main.dispatch delivered fbb3e230-f39d-4d30-a8da-a3d21096ea99 to worker[181] qsize 0
2022-08-28 03:01:57,421 DEBUG awx.main.dispatch task fbb3e230-f39d-4d30-a8da-a3d21096ea99 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:01:57,448 DEBUG awx.main.dispatch task fbb3e230-f39d-4d30-a8da-a3d21096ea99 is finished
2022-08-28 03:01:58,454 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dc96d359-3e71-4650-b9be-6aaec88f0f82, queue=awx_private_queue)
[2022-08-28 03:01:58,454: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dc96d359-3e71-4650-b9be-6aaec88f0f82, queue=awx_private_queue)
2022-08-28 03:01:58,506 DEBUG awx.main.dispatch delivered dc96d359-3e71-4650-b9be-6aaec88f0f82 to worker[182] qsize 0
2022-08-28 03:01:58,509 DEBUG awx.main.dispatch task dc96d359-3e71-4650-b9be-6aaec88f0f82 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:01:58,526 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:01:58,529 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:01:28.536439+00:00
2022-08-28 03:01:58,546 DEBUG awx.main.dispatch task dc96d359-3e71-4650-b9be-6aaec88f0f82 is finished
2022-08-28 03:02:00,718 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2fb6581c-ae4b-436a-aa95-0babd96d38ff, queue=awx_private_queue)
[2022-08-28 03:02:00,718: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2fb6581c-ae4b-436a-aa95-0babd96d38ff, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:02:00,733 DEBUG awx.main.dispatch delivered 2fb6581c-ae4b-436a-aa95-0babd96d38ff to worker[182] qsize 0
2022-08-28 03:02:00,920 DEBUG awx.main.dispatch task 2fb6581c-ae4b-436a-aa95-0babd96d38ff starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:02:00,922 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:02:00,975 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:02:01,036 DEBUG awx.main.dispatch task 2fb6581c-ae4b-436a-aa95-0babd96d38ff is finished
2022-08-28 03:02:09,378 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:02:09,384 DEBUG awx.main.dispatch task 0bc3a5d4-d391-47f8-a35c-9996dfd4ab96 is finished
2022-08-28 03:02:09,378 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection timed out
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection timed out
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:02:09,387 DEBUG awx.main.dispatch task 23de2f42-64cc-4887-8388-44ee611f0ec5 is finished
2022-08-28 03:02:20,722 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f53ef156-57a2-4ee6-8da2-3301ad394b07, queue=awx_private_queue)
[2022-08-28 03:02:20,722: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f53ef156-57a2-4ee6-8da2-3301ad394b07, queue=awx_private_queue)
2022-08-28 03:02:20,738 DEBUG awx.main.dispatch delivered f53ef156-57a2-4ee6-8da2-3301ad394b07 to worker[183] qsize 0
2022-08-28 03:02:20,740 DEBUG awx.main.dispatch task f53ef156-57a2-4ee6-8da2-3301ad394b07 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:02:20,741 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:02:20,769 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:02:20,807 DEBUG awx.main.dispatch task f53ef156-57a2-4ee6-8da2-3301ad394b07 is finished
2022-08-28 03:02:28,455 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a646bd6c-0706-4310-bbe5-a492a2d61f38, queue=awx_private_queue)
[2022-08-28 03:02:28,455: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a646bd6c-0706-4310-bbe5-a492a2d61f38, queue=awx_private_queue)
2022-08-28 03:02:28,473 DEBUG awx.main.dispatch delivered a646bd6c-0706-4310-bbe5-a492a2d61f38 to worker[182] qsize 0
2022-08-28 03:02:28,476 DEBUG awx.main.dispatch task a646bd6c-0706-4310-bbe5-a492a2d61f38 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:02:28,519 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:02:28,527 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:01:58.526264+00:00
2022-08-28 03:02:28,585 DEBUG awx.main.dispatch task a646bd6c-0706-4310-bbe5-a492a2d61f38 is finished
2022-08-28 03:02:40,723 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e0c2f185-319e-42b0-81b6-e738f20678e8, queue=awx_private_queue)
[2022-08-28 03:02:40,723: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e0c2f185-319e-42b0-81b6-e738f20678e8, queue=awx_private_queue)
2022-08-28 03:02:40,739 DEBUG awx.main.dispatch delivered e0c2f185-319e-42b0-81b6-e738f20678e8 to worker[183] qsize 0
2022-08-28 03:02:40,742 DEBUG awx.main.dispatch task e0c2f185-319e-42b0-81b6-e738f20678e8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:02:40,743 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:02:40,765 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:02:40,847 DEBUG awx.main.dispatch task e0c2f185-319e-42b0-81b6-e738f20678e8 is finished
2022-08-28 03:02:56,343 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e7a20e34-03f6-4eea-a8b4-b23732483170, queue=onap-awx-0)
[2022-08-28 03:02:56,343: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e7a20e34-03f6-4eea-a8b4-b23732483170, queue=onap-awx-0)
2022-08-28 03:02:56,371 DEBUG awx.main.dispatch delivered e7a20e34-03f6-4eea-a8b4-b23732483170 to worker[181] qsize 0
2022-08-28 03:02:56,373 DEBUG awx.main.dispatch task e7a20e34-03f6-4eea-a8b4-b23732483170 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:02:56,375 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:02:56,402 DEBUG awx.main.dispatch task e7a20e34-03f6-4eea-a8b4-b23732483170 is finished
2022-08-28 03:02:57,400 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(729c9fb9-53ea-4fdb-bc68-1fb5fe375568, queue=onap-awx-0)
[2022-08-28 03:02:57,400: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(729c9fb9-53ea-4fdb-bc68-1fb5fe375568, queue=onap-awx-0)
2022-08-28 03:02:57,416 DEBUG awx.main.dispatch delivered 729c9fb9-53ea-4fdb-bc68-1fb5fe375568 to worker[181] qsize 0
2022-08-28 03:02:57,419 DEBUG awx.main.dispatch task 729c9fb9-53ea-4fdb-bc68-1fb5fe375568 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:02:57,435 DEBUG awx.main.dispatch task 729c9fb9-53ea-4fdb-bc68-1fb5fe375568 is finished
2022-08-28 03:02:58,454 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(02210ba1-2a85-47c9-997e-a3f5321df6ed, queue=awx_private_queue)
[2022-08-28 03:02:58,454: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(02210ba1-2a85-47c9-997e-a3f5321df6ed, queue=awx_private_queue)
2022-08-28 03:02:58,470 DEBUG awx.main.dispatch delivered 02210ba1-2a85-47c9-997e-a3f5321df6ed to worker[182] qsize 0
2022-08-28 03:02:58,472 DEBUG awx.main.dispatch task 02210ba1-2a85-47c9-997e-a3f5321df6ed starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:02:58,510 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:02:58,527 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:02:28.519675+00:00
2022-08-28 03:02:58,572 DEBUG awx.main.dispatch task 02210ba1-2a85-47c9-997e-a3f5321df6ed is finished
2022-08-28 03:03:00,723 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d4674dd2-dbdf-48ad-acf4-ca0ff522ca74, queue=awx_private_queue)
[2022-08-28 03:03:00,723: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d4674dd2-dbdf-48ad-acf4-ca0ff522ca74, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:03:00,793 DEBUG awx.main.dispatch delivered d4674dd2-dbdf-48ad-acf4-ca0ff522ca74 to worker[183] qsize 0
2022-08-28 03:03:00,991 DEBUG awx.main.dispatch task d4674dd2-dbdf-48ad-acf4-ca0ff522ca74 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:03:00,993 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:03:01,032 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:03:01,094 DEBUG awx.main.dispatch task d4674dd2-dbdf-48ad-acf4-ca0ff522ca74 is finished
2022-08-28 03:03:20,733 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6417319f-731d-45cd-b6b1-6d60658d8ce4, queue=awx_private_queue)
[2022-08-28 03:03:20,733: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6417319f-731d-45cd-b6b1-6d60658d8ce4, queue=awx_private_queue)
2022-08-28 03:03:20,748 DEBUG awx.main.dispatch delivered 6417319f-731d-45cd-b6b1-6d60658d8ce4 to worker[183] qsize 0
2022-08-28 03:03:20,750 DEBUG awx.main.dispatch task 6417319f-731d-45cd-b6b1-6d60658d8ce4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:03:20,752 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:03:20,779 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:03:20,816 DEBUG awx.main.dispatch task 6417319f-731d-45cd-b6b1-6d60658d8ce4 is finished
2022-08-28 03:03:28,454 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(87d6dd5e-9d0e-4d63-8a84-c5059ab5e04a, queue=awx_private_queue)
[2022-08-28 03:03:28,454: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(87d6dd5e-9d0e-4d63-8a84-c5059ab5e04a, queue=awx_private_queue)
2022-08-28 03:03:28,469 DEBUG awx.main.dispatch delivered 87d6dd5e-9d0e-4d63-8a84-c5059ab5e04a to worker[183] qsize 0
2022-08-28 03:03:28,472 DEBUG awx.main.dispatch task 87d6dd5e-9d0e-4d63-8a84-c5059ab5e04a starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:03:28,488 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:03:28,491 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:02:58.510885+00:00
2022-08-28 03:03:28,522 DEBUG awx.main.dispatch task 87d6dd5e-9d0e-4d63-8a84-c5059ab5e04a is finished
2022-08-28 03:03:40,736 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4ca7c46d-5b74-4529-95b5-57dcdeff94a4, queue=awx_private_queue)
[2022-08-28 03:03:40,736: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4ca7c46d-5b74-4529-95b5-57dcdeff94a4, queue=awx_private_queue)
2022-08-28 03:03:40,752 DEBUG awx.main.dispatch delivered 4ca7c46d-5b74-4529-95b5-57dcdeff94a4 to worker[181] qsize 0
2022-08-28 03:03:40,754 DEBUG awx.main.dispatch task 4ca7c46d-5b74-4529-95b5-57dcdeff94a4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:03:40,756 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:03:40,828 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:03:40,867 DEBUG awx.main.dispatch task 4ca7c46d-5b74-4529-95b5-57dcdeff94a4 is finished
2022-08-28 03:03:56,344 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f9e936b0-5c14-47d0-9751-7926c0e5b440, queue=onap-awx-0)
[2022-08-28 03:03:56,344: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f9e936b0-5c14-47d0-9751-7926c0e5b440, queue=onap-awx-0)
2022-08-28 03:03:56,373 DEBUG awx.main.dispatch delivered f9e936b0-5c14-47d0-9751-7926c0e5b440 to worker[181] qsize 0
2022-08-28 03:03:56,376 DEBUG awx.main.dispatch task f9e936b0-5c14-47d0-9751-7926c0e5b440 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:03:56,378 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:03:56,407 DEBUG awx.main.dispatch task f9e936b0-5c14-47d0-9751-7926c0e5b440 is finished
2022-08-28 03:03:57,400 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f9ef95e8-a392-4f91-99e6-47b05c39a3f3, queue=onap-awx-0)
[2022-08-28 03:03:57,400: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f9ef95e8-a392-4f91-99e6-47b05c39a3f3, queue=onap-awx-0)
2022-08-28 03:03:57,414 DEBUG awx.main.dispatch delivered f9ef95e8-a392-4f91-99e6-47b05c39a3f3 to worker[181] qsize 0
2022-08-28 03:03:57,416 DEBUG awx.main.dispatch task f9ef95e8-a392-4f91-99e6-47b05c39a3f3 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:03:57,447 DEBUG awx.main.dispatch task f9ef95e8-a392-4f91-99e6-47b05c39a3f3 is finished
2022-08-28 03:03:58,454 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(06389d3a-d454-4e9f-82cc-e92a11e3917a, queue=awx_private_queue)
[2022-08-28 03:03:58,454: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(06389d3a-d454-4e9f-82cc-e92a11e3917a, queue=awx_private_queue)
2022-08-28 03:03:58,470 DEBUG awx.main.dispatch delivered 06389d3a-d454-4e9f-82cc-e92a11e3917a to worker[180] qsize 0
2022-08-28 03:03:58,473 DEBUG awx.main.dispatch task 06389d3a-d454-4e9f-82cc-e92a11e3917a starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:03:58,487 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:03:58,491 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:03:28.488503+00:00
2022-08-28 03:03:58,509 DEBUG awx.main.dispatch task 06389d3a-d454-4e9f-82cc-e92a11e3917a is finished
2022-08-28 03:04:00,736 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(93e665c4-af9b-41bf-a692-4a5ad6a0fa10, queue=awx_private_queue)
[2022-08-28 03:04:00,736: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(93e665c4-af9b-41bf-a692-4a5ad6a0fa10, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:04:00,771 DEBUG awx.main.dispatch delivered 93e665c4-af9b-41bf-a692-4a5ad6a0fa10 to worker[181] qsize 0
2022-08-28 03:04:00,774 DEBUG awx.main.dispatch task 93e665c4-af9b-41bf-a692-4a5ad6a0fa10 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:04:00,775 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:04:00,802 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:04:00,852 DEBUG awx.main.dispatch task 93e665c4-af9b-41bf-a692-4a5ad6a0fa10 is finished
2022-08-28 03:04:20,746 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(badd4003-6e1f-4206-92e9-fadafe88150b, queue=awx_private_queue)
[2022-08-28 03:04:20,746: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(badd4003-6e1f-4206-92e9-fadafe88150b, queue=awx_private_queue)
2022-08-28 03:04:20,762 DEBUG awx.main.dispatch delivered badd4003-6e1f-4206-92e9-fadafe88150b to worker[182] qsize 0
2022-08-28 03:04:20,947 DEBUG awx.main.dispatch task badd4003-6e1f-4206-92e9-fadafe88150b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:04:20,949 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:04:20,974 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:04:21,022 DEBUG awx.main.dispatch task badd4003-6e1f-4206-92e9-fadafe88150b is finished
2022-08-28 03:04:28,454 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(95496b4d-ec0c-47df-b155-a1c489fbc834, queue=awx_private_queue)
[2022-08-28 03:04:28,454: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(95496b4d-ec0c-47df-b155-a1c489fbc834, queue=awx_private_queue)
2022-08-28 03:04:28,470 DEBUG awx.main.dispatch delivered 95496b4d-ec0c-47df-b155-a1c489fbc834 to worker[180] qsize 0
2022-08-28 03:04:28,472 DEBUG awx.main.dispatch task 95496b4d-ec0c-47df-b155-a1c489fbc834 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:04:28,546 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:04:28,552 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:03:58.487629+00:00
2022-08-28 03:04:28,650 DEBUG awx.main.dispatch task 95496b4d-ec0c-47df-b155-a1c489fbc834 is finished
2022-08-28 03:04:40,751 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b2d85ff2-33bb-44f9-a1cc-97e4eb2f6970, queue=awx_private_queue)
[2022-08-28 03:04:40,751: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b2d85ff2-33bb-44f9-a1cc-97e4eb2f6970, queue=awx_private_queue)
2022-08-28 03:04:40,788 DEBUG awx.main.dispatch delivered b2d85ff2-33bb-44f9-a1cc-97e4eb2f6970 to worker[180] qsize 0
2022-08-28 03:04:40,792 DEBUG awx.main.dispatch task b2d85ff2-33bb-44f9-a1cc-97e4eb2f6970 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:04:40,795 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:04:40,852 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:04:40,928 DEBUG awx.main.dispatch task b2d85ff2-33bb-44f9-a1cc-97e4eb2f6970 is finished
2022-08-28 03:04:56,347 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(19767f9f-1c76-4632-94b6-0e16f80e58b8, queue=onap-awx-0)
[2022-08-28 03:04:56,347: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(19767f9f-1c76-4632-94b6-0e16f80e58b8, queue=onap-awx-0)
2022-08-28 03:04:56,383 DEBUG awx.main.dispatch delivered 19767f9f-1c76-4632-94b6-0e16f80e58b8 to worker[180] qsize 0
2022-08-28 03:04:56,385 DEBUG awx.main.dispatch task 19767f9f-1c76-4632-94b6-0e16f80e58b8 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:04:56,386 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:04:56,422 DEBUG awx.main.dispatch task 19767f9f-1c76-4632-94b6-0e16f80e58b8 is finished
2022-08-28 03:04:57,400 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(e8df9abd-e681-4638-b80e-2477f5520f8f, queue=onap-awx-0)
[2022-08-28 03:04:57,400: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(e8df9abd-e681-4638-b80e-2477f5520f8f, queue=onap-awx-0)
2022-08-28 03:04:57,417 DEBUG awx.main.dispatch delivered e8df9abd-e681-4638-b80e-2477f5520f8f to worker[181] qsize 0
2022-08-28 03:04:57,420 DEBUG awx.main.dispatch task e8df9abd-e681-4638-b80e-2477f5520f8f starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:04:57,439 DEBUG awx.main.dispatch task e8df9abd-e681-4638-b80e-2477f5520f8f is finished
2022-08-28 03:04:58,455 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f180d3f3-0099-4ac5-af7d-7ac255e77df2, queue=awx_private_queue)
[2022-08-28 03:04:58,455: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f180d3f3-0099-4ac5-af7d-7ac255e77df2, queue=awx_private_queue)
2022-08-28 03:04:58,470 DEBUG awx.main.dispatch delivered f180d3f3-0099-4ac5-af7d-7ac255e77df2 to worker[182] qsize 0
2022-08-28 03:04:58,473 DEBUG awx.main.dispatch task f180d3f3-0099-4ac5-af7d-7ac255e77df2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:04:58,512 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:04:58,518 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:04:28.546805+00:00
2022-08-28 03:04:58,552 DEBUG awx.main.dispatch task f180d3f3-0099-4ac5-af7d-7ac255e77df2 is finished
2022-08-28 03:05:00,749 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7797f1b3-04bb-40ca-bfef-44d8f96c6f02, queue=awx_private_queue)
[2022-08-28 03:05:00,749: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7797f1b3-04bb-40ca-bfef-44d8f96c6f02, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:05:00,765 DEBUG awx.main.dispatch delivered 7797f1b3-04bb-40ca-bfef-44d8f96c6f02 to worker[182] qsize 0
2022-08-28 03:05:00,768 DEBUG awx.main.dispatch task 7797f1b3-04bb-40ca-bfef-44d8f96c6f02 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:05:00,769 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:05:00,815 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:05:00,855 DEBUG awx.main.dispatch task 7797f1b3-04bb-40ca-bfef-44d8f96c6f02 is finished
2022-08-28 03:05:20,750 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5ec000b9-3559-4aaa-89d0-825ff20aba56, queue=awx_private_queue)
[2022-08-28 03:05:20,750: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5ec000b9-3559-4aaa-89d0-825ff20aba56, queue=awx_private_queue)
2022-08-28 03:05:20,807 DEBUG awx.main.dispatch delivered 5ec000b9-3559-4aaa-89d0-825ff20aba56 to worker[182] qsize 0
2022-08-28 03:05:20,987 DEBUG awx.main.dispatch task 5ec000b9-3559-4aaa-89d0-825ff20aba56 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:05:20,990 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:05:21,032 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:05:21,095 DEBUG awx.main.dispatch task 5ec000b9-3559-4aaa-89d0-825ff20aba56 is finished
2022-08-28 03:05:28,456 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(da6d5ff2-ff57-4284-b323-bcc1145c7d52, queue=awx_private_queue)
[2022-08-28 03:05:28,456: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(da6d5ff2-ff57-4284-b323-bcc1145c7d52, queue=awx_private_queue)
2022-08-28 03:05:28,472 DEBUG awx.main.dispatch delivered da6d5ff2-ff57-4284-b323-bcc1145c7d52 to worker[183] qsize 0
2022-08-28 03:05:28,475 DEBUG awx.main.dispatch task da6d5ff2-ff57-4284-b323-bcc1145c7d52 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:05:28,489 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:05:28,493 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:04:58.513155+00:00
2022-08-28 03:05:28,510 DEBUG awx.main.dispatch task da6d5ff2-ff57-4284-b323-bcc1145c7d52 is finished
2022-08-28 03:05:40,753 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a95ac74b-b2a8-457d-8d01-81bfe0e3db05, queue=awx_private_queue)
[2022-08-28 03:05:40,753: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a95ac74b-b2a8-457d-8d01-81bfe0e3db05, queue=awx_private_queue)
2022-08-28 03:05:40,770 DEBUG awx.main.dispatch delivered a95ac74b-b2a8-457d-8d01-81bfe0e3db05 to worker[181] qsize 0
2022-08-28 03:05:40,772 DEBUG awx.main.dispatch task a95ac74b-b2a8-457d-8d01-81bfe0e3db05 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:05:40,774 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:05:40,810 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:05:40,897 DEBUG awx.main.dispatch task a95ac74b-b2a8-457d-8d01-81bfe0e3db05 is finished
2022-08-28 03:05:56,351 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d7fad0e5-d544-42c2-851f-ee06c4605946, queue=onap-awx-0)
[2022-08-28 03:05:56,351: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d7fad0e5-d544-42c2-851f-ee06c4605946, queue=onap-awx-0)
2022-08-28 03:05:56,379 DEBUG awx.main.dispatch delivered d7fad0e5-d544-42c2-851f-ee06c4605946 to worker[182] qsize 0
2022-08-28 03:05:56,381 DEBUG awx.main.dispatch task d7fad0e5-d544-42c2-851f-ee06c4605946 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:05:56,383 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:05:56,414 DEBUG awx.main.dispatch task d7fad0e5-d544-42c2-851f-ee06c4605946 is finished
2022-08-28 03:05:57,400 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(fa8c254f-1b1b-4cc9-8dbc-d25601ba9c9d, queue=onap-awx-0)
[2022-08-28 03:05:57,400: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(fa8c254f-1b1b-4cc9-8dbc-d25601ba9c9d, queue=onap-awx-0)
2022-08-28 03:05:57,420 DEBUG awx.main.dispatch delivered fa8c254f-1b1b-4cc9-8dbc-d25601ba9c9d to worker[182] qsize 0
2022-08-28 03:05:57,422 DEBUG awx.main.dispatch task fa8c254f-1b1b-4cc9-8dbc-d25601ba9c9d starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:05:57,439 DEBUG awx.main.dispatch task fa8c254f-1b1b-4cc9-8dbc-d25601ba9c9d is finished
2022-08-28 03:05:58,455 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(93a70df2-65e4-484e-952e-85310a2179e3, queue=awx_private_queue)
[2022-08-28 03:05:58,455: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(93a70df2-65e4-484e-952e-85310a2179e3, queue=awx_private_queue)
2022-08-28 03:05:58,471 DEBUG awx.main.dispatch delivered 93a70df2-65e4-484e-952e-85310a2179e3 to worker[183] qsize 0
2022-08-28 03:05:58,473 DEBUG awx.main.dispatch task 93a70df2-65e4-484e-952e-85310a2179e3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:05:58,496 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:05:58,507 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:05:28.489734+00:00
2022-08-28 03:05:58,524 DEBUG awx.main.dispatch task 93a70df2-65e4-484e-952e-85310a2179e3 is finished
2022-08-28 03:06:00,753 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fd7d5671-b958-47ba-a07a-777550dcaac9, queue=awx_private_queue)
[2022-08-28 03:06:00,753: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fd7d5671-b958-47ba-a07a-777550dcaac9, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:06:00,767 DEBUG awx.main.dispatch delivered fd7d5671-b958-47ba-a07a-777550dcaac9 to worker[183] qsize 0
2022-08-28 03:06:00,770 DEBUG awx.main.dispatch task fd7d5671-b958-47ba-a07a-777550dcaac9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:06:00,771 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:06:00,796 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:06:00,833 DEBUG awx.main.dispatch task fd7d5671-b958-47ba-a07a-777550dcaac9 is finished
2022-08-28 03:06:20,757 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3822acbb-8875-483e-9740-d24d41a49d92, queue=awx_private_queue)
[2022-08-28 03:06:20,757: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3822acbb-8875-483e-9740-d24d41a49d92, queue=awx_private_queue)
2022-08-28 03:06:20,791 DEBUG awx.main.dispatch delivered 3822acbb-8875-483e-9740-d24d41a49d92 to worker[180] qsize 0
2022-08-28 03:06:20,793 DEBUG awx.main.dispatch task 3822acbb-8875-483e-9740-d24d41a49d92 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:06:21,004 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:06:21,016 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:06:21,069 DEBUG awx.main.dispatch task 3822acbb-8875-483e-9740-d24d41a49d92 is finished
2022-08-28 03:06:28,455 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ed8105e8-6345-4fc1-b4cf-687cf992a734, queue=awx_private_queue)
[2022-08-28 03:06:28,455: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ed8105e8-6345-4fc1-b4cf-687cf992a734, queue=awx_private_queue)
2022-08-28 03:06:28,470 DEBUG awx.main.dispatch delivered ed8105e8-6345-4fc1-b4cf-687cf992a734 to worker[182] qsize 0
2022-08-28 03:06:28,473 DEBUG awx.main.dispatch task ed8105e8-6345-4fc1-b4cf-687cf992a734 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:06:28,485 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:06:28,498 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:05:58.496484+00:00
2022-08-28 03:06:28,528 DEBUG awx.main.dispatch task ed8105e8-6345-4fc1-b4cf-687cf992a734 is finished
2022-08-28 03:06:40,757 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(57091cc2-c88a-4c09-8299-a80d1af8657f, queue=awx_private_queue)
[2022-08-28 03:06:40,757: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(57091cc2-c88a-4c09-8299-a80d1af8657f, queue=awx_private_queue)
2022-08-28 03:06:40,771 DEBUG awx.main.dispatch delivered 57091cc2-c88a-4c09-8299-a80d1af8657f to worker[181] qsize 0
2022-08-28 03:06:40,773 DEBUG awx.main.dispatch task 57091cc2-c88a-4c09-8299-a80d1af8657f starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:06:40,775 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:06:40,845 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:06:40,920 DEBUG awx.main.dispatch task 57091cc2-c88a-4c09-8299-a80d1af8657f is finished
2022-08-28 03:06:56,357 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(dc797d3e-e3a5-4724-abd1-ccc44c9337fe, queue=onap-awx-0)
[2022-08-28 03:06:56,357: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(dc797d3e-e3a5-4724-abd1-ccc44c9337fe, queue=onap-awx-0)
2022-08-28 03:06:56,386 DEBUG awx.main.dispatch delivered dc797d3e-e3a5-4724-abd1-ccc44c9337fe to worker[180] qsize 0
2022-08-28 03:06:56,388 DEBUG awx.main.dispatch task dc797d3e-e3a5-4724-abd1-ccc44c9337fe starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:06:56,389 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:06:56,426 DEBUG awx.main.dispatch task dc797d3e-e3a5-4724-abd1-ccc44c9337fe is finished
2022-08-28 03:06:57,400 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f794b58d-8d0d-4d92-9dcc-3c5bacc4f048, queue=onap-awx-0)
[2022-08-28 03:06:57,400: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f794b58d-8d0d-4d92-9dcc-3c5bacc4f048, queue=onap-awx-0)
2022-08-28 03:06:57,416 DEBUG awx.main.dispatch delivered f794b58d-8d0d-4d92-9dcc-3c5bacc4f048 to worker[180] qsize 0
2022-08-28 03:06:57,418 DEBUG awx.main.dispatch task f794b58d-8d0d-4d92-9dcc-3c5bacc4f048 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:06:57,434 DEBUG awx.main.dispatch task f794b58d-8d0d-4d92-9dcc-3c5bacc4f048 is finished
2022-08-28 03:06:58,455 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(25adb06a-50a1-4f68-8c70-b5a1a2f1b729, queue=awx_private_queue)
[2022-08-28 03:06:58,455: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(25adb06a-50a1-4f68-8c70-b5a1a2f1b729, queue=awx_private_queue)
2022-08-28 03:06:58,470 DEBUG awx.main.dispatch delivered 25adb06a-50a1-4f68-8c70-b5a1a2f1b729 to worker[181] qsize 0
2022-08-28 03:06:58,472 DEBUG awx.main.dispatch task 25adb06a-50a1-4f68-8c70-b5a1a2f1b729 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:06:58,484 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:06:58,487 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:06:28.486009+00:00
2022-08-28 03:06:58,504 DEBUG awx.main.dispatch task 25adb06a-50a1-4f68-8c70-b5a1a2f1b729 is finished
2022-08-28 03:07:00,757 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e1ef8e68-e3be-4d50-ae35-31d60e876fe7, queue=awx_private_queue)
[2022-08-28 03:07:00,757: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e1ef8e68-e3be-4d50-ae35-31d60e876fe7, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:07:00,773 DEBUG awx.main.dispatch delivered e1ef8e68-e3be-4d50-ae35-31d60e876fe7 to worker[182] qsize 0
2022-08-28 03:07:00,775 DEBUG awx.main.dispatch task e1ef8e68-e3be-4d50-ae35-31d60e876fe7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:07:00,777 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:07:00,819 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:07:00,856 DEBUG awx.main.dispatch task e1ef8e68-e3be-4d50-ae35-31d60e876fe7 is finished
2022-08-28 03:07:20,757 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(93841b6c-b920-4b15-9192-f7c29e1d9cff, queue=awx_private_queue)
[2022-08-28 03:07:20,757: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(93841b6c-b920-4b15-9192-f7c29e1d9cff, queue=awx_private_queue)
2022-08-28 03:07:20,789 DEBUG awx.main.dispatch delivered 93841b6c-b920-4b15-9192-f7c29e1d9cff to worker[181] qsize 0
2022-08-28 03:07:20,791 DEBUG awx.main.dispatch task 93841b6c-b920-4b15-9192-f7c29e1d9cff starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:07:20,793 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:07:20,822 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:07:20,868 DEBUG awx.main.dispatch task 93841b6c-b920-4b15-9192-f7c29e1d9cff is finished
2022-08-28 03:07:28,456 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(57926594-5c26-4099-a6f0-a27107571265, queue=awx_private_queue)
[2022-08-28 03:07:28,456: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(57926594-5c26-4099-a6f0-a27107571265, queue=awx_private_queue)
2022-08-28 03:07:28,474 DEBUG awx.main.dispatch delivered 57926594-5c26-4099-a6f0-a27107571265 to worker[183] qsize 0
2022-08-28 03:07:28,662 DEBUG awx.main.dispatch task 57926594-5c26-4099-a6f0-a27107571265 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:07:28,677 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:07:28,682 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:06:58.484347+00:00
2022-08-28 03:07:28,707 DEBUG awx.main.dispatch task 57926594-5c26-4099-a6f0-a27107571265 is finished
2022-08-28 03:07:40,757 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7c3cf521-5e06-40c9-a64d-9c3966cc63dc, queue=awx_private_queue)
[2022-08-28 03:07:40,757: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7c3cf521-5e06-40c9-a64d-9c3966cc63dc, queue=awx_private_queue)
2022-08-28 03:07:40,773 DEBUG awx.main.dispatch delivered 7c3cf521-5e06-40c9-a64d-9c3966cc63dc to worker[181] qsize 0
2022-08-28 03:07:40,775 DEBUG awx.main.dispatch task 7c3cf521-5e06-40c9-a64d-9c3966cc63dc starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:07:40,777 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:07:40,866 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:07:40,905 DEBUG awx.main.dispatch task 7c3cf521-5e06-40c9-a64d-9c3966cc63dc is finished
2022-08-28 03:07:56,362 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6bc512a4-fa1f-464d-a7fa-bd0c83b95de0, queue=onap-awx-0)
[2022-08-28 03:07:56,362: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6bc512a4-fa1f-464d-a7fa-bd0c83b95de0, queue=onap-awx-0)
2022-08-28 03:07:56,405 DEBUG awx.main.dispatch delivered 6bc512a4-fa1f-464d-a7fa-bd0c83b95de0 to worker[183] qsize 0
2022-08-28 03:07:56,408 DEBUG awx.main.dispatch task 6bc512a4-fa1f-464d-a7fa-bd0c83b95de0 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:07:56,409 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:07:56,470 DEBUG awx.main.dispatch task 6bc512a4-fa1f-464d-a7fa-bd0c83b95de0 is finished
2022-08-28 03:07:57,400 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(eb744ae4-4d02-4155-8148-83c0573be52e, queue=onap-awx-0)
[2022-08-28 03:07:57,400: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(eb744ae4-4d02-4155-8148-83c0573be52e, queue=onap-awx-0)
2022-08-28 03:07:57,417 DEBUG awx.main.dispatch delivered eb744ae4-4d02-4155-8148-83c0573be52e to worker[183] qsize 0
2022-08-28 03:07:57,420 DEBUG awx.main.dispatch task eb744ae4-4d02-4155-8148-83c0573be52e starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:07:57,437 DEBUG awx.main.dispatch task eb744ae4-4d02-4155-8148-83c0573be52e is finished
2022-08-28 03:07:58,455 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(201bb73d-ab2a-4461-a281-f74208260269, queue=awx_private_queue)
[2022-08-28 03:07:58,455: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(201bb73d-ab2a-4461-a281-f74208260269, queue=awx_private_queue)
2022-08-28 03:07:58,470 DEBUG awx.main.dispatch delivered 201bb73d-ab2a-4461-a281-f74208260269 to worker[183] qsize 0
2022-08-28 03:07:58,473 DEBUG awx.main.dispatch task 201bb73d-ab2a-4461-a281-f74208260269 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:07:58,496 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:07:58,502 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:07:28.678024+00:00
2022-08-28 03:07:58,559 DEBUG awx.main.dispatch task 201bb73d-ab2a-4461-a281-f74208260269 is finished
2022-08-28 03:08:00,757 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5cab7ff8-cd53-4d87-bad7-6273cd227b1c, queue=awx_private_queue)
[2022-08-28 03:08:00,757: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5cab7ff8-cd53-4d87-bad7-6273cd227b1c, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:08:00,781 DEBUG awx.main.dispatch delivered 5cab7ff8-cd53-4d87-bad7-6273cd227b1c to worker[181] qsize 0
2022-08-28 03:08:00,783 DEBUG awx.main.dispatch task 5cab7ff8-cd53-4d87-bad7-6273cd227b1c starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:08:00,785 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:08:00,804 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:08:00,849 DEBUG awx.main.dispatch task 5cab7ff8-cd53-4d87-bad7-6273cd227b1c is finished
2022-08-28 03:08:20,764 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9a1c96bd-5289-4e37-827c-eb1300e9b2da, queue=awx_private_queue)
[2022-08-28 03:08:20,764: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9a1c96bd-5289-4e37-827c-eb1300e9b2da, queue=awx_private_queue)
2022-08-28 03:08:20,782 DEBUG awx.main.dispatch delivered 9a1c96bd-5289-4e37-827c-eb1300e9b2da to worker[180] qsize 0
2022-08-28 03:08:20,785 DEBUG awx.main.dispatch task 9a1c96bd-5289-4e37-827c-eb1300e9b2da starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:08:20,789 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:08:20,814 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:08:20,847 DEBUG awx.main.dispatch task 9a1c96bd-5289-4e37-827c-eb1300e9b2da is finished
2022-08-28 03:08:28,456 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1a5372f2-e936-4fa0-8866-2e3fbe096cfc, queue=awx_private_queue)
[2022-08-28 03:08:28,456: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1a5372f2-e936-4fa0-8866-2e3fbe096cfc, queue=awx_private_queue)
2022-08-28 03:08:28,473 DEBUG awx.main.dispatch delivered 1a5372f2-e936-4fa0-8866-2e3fbe096cfc to worker[183] qsize 0
2022-08-28 03:08:28,476 DEBUG awx.main.dispatch task 1a5372f2-e936-4fa0-8866-2e3fbe096cfc starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:08:28,499 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:08:28,506 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:07:58.497090+00:00
2022-08-28 03:08:28,555 DEBUG awx.main.dispatch task 1a5372f2-e936-4fa0-8866-2e3fbe096cfc is finished
2022-08-28 03:08:40,767 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(53258434-52b3-4c7a-a088-fefac4f734c1, queue=awx_private_queue)
[2022-08-28 03:08:40,767: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(53258434-52b3-4c7a-a088-fefac4f734c1, queue=awx_private_queue)
2022-08-28 03:08:40,783 DEBUG awx.main.dispatch delivered 53258434-52b3-4c7a-a088-fefac4f734c1 to worker[181] qsize 0
2022-08-28 03:08:41,001 DEBUG awx.main.dispatch task 53258434-52b3-4c7a-a088-fefac4f734c1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:08:41,003 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:08:41,100 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:08:41,321 DEBUG awx.main.dispatch task 53258434-52b3-4c7a-a088-fefac4f734c1 is finished
2022-08-28 03:08:56,369 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(7aa36918-09cc-43d3-aba9-b70da3ea4ea3, queue=onap-awx-0)
[2022-08-28 03:08:56,369: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(7aa36918-09cc-43d3-aba9-b70da3ea4ea3, queue=onap-awx-0)
2022-08-28 03:08:56,459 DEBUG awx.main.dispatch delivered 7aa36918-09cc-43d3-aba9-b70da3ea4ea3 to worker[183] qsize 0
2022-08-28 03:08:56,461 DEBUG awx.main.dispatch task 7aa36918-09cc-43d3-aba9-b70da3ea4ea3 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:08:56,463 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:08:56,637 DEBUG awx.main.dispatch task 7aa36918-09cc-43d3-aba9-b70da3ea4ea3 is finished
2022-08-28 03:08:57,400 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(be824531-3a05-4fe1-99ed-4896027432b4, queue=onap-awx-0)
[2022-08-28 03:08:57,400: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(be824531-3a05-4fe1-99ed-4896027432b4, queue=onap-awx-0)
2022-08-28 03:08:57,421 DEBUG awx.main.dispatch delivered be824531-3a05-4fe1-99ed-4896027432b4 to worker[183] qsize 0
2022-08-28 03:08:57,424 DEBUG awx.main.dispatch task be824531-3a05-4fe1-99ed-4896027432b4 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:08:57,490 DEBUG awx.main.dispatch task be824531-3a05-4fe1-99ed-4896027432b4 is finished
2022-08-28 03:08:58,455 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(34fe27d6-f1e7-413e-8dc6-1847a7c3d1c8, queue=awx_private_queue)
[2022-08-28 03:08:58,455: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(34fe27d6-f1e7-413e-8dc6-1847a7c3d1c8, queue=awx_private_queue)
2022-08-28 03:08:58,471 DEBUG awx.main.dispatch delivered 34fe27d6-f1e7-413e-8dc6-1847a7c3d1c8 to worker[180] qsize 0
2022-08-28 03:08:58,473 DEBUG awx.main.dispatch task 34fe27d6-f1e7-413e-8dc6-1847a7c3d1c8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:08:58,507 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:08:58,532 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:08:28.499659+00:00
2022-08-28 03:08:58,624 DEBUG awx.main.dispatch task 34fe27d6-f1e7-413e-8dc6-1847a7c3d1c8 is finished
2022-08-28 03:09:00,766 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7cc09ea5-f688-42fa-80d0-66c6dda618a8, queue=awx_private_queue)
[2022-08-28 03:09:00,766: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7cc09ea5-f688-42fa-80d0-66c6dda618a8, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:09:00,782 DEBUG awx.main.dispatch delivered 7cc09ea5-f688-42fa-80d0-66c6dda618a8 to worker[180] qsize 0
2022-08-28 03:09:00,784 DEBUG awx.main.dispatch task 7cc09ea5-f688-42fa-80d0-66c6dda618a8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:09:00,786 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:09:01,082 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:09:01,239 DEBUG awx.main.dispatch task 7cc09ea5-f688-42fa-80d0-66c6dda618a8 is finished
2022-08-28 03:09:20,779 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(03510177-70cf-4c85-a77e-6604afe34d24, queue=awx_private_queue)
[2022-08-28 03:09:20,779: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(03510177-70cf-4c85-a77e-6604afe34d24, queue=awx_private_queue)
2022-08-28 03:09:20,803 DEBUG awx.main.dispatch delivered 03510177-70cf-4c85-a77e-6604afe34d24 to worker[181] qsize 0
2022-08-28 03:09:20,805 DEBUG awx.main.dispatch task 03510177-70cf-4c85-a77e-6604afe34d24 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:09:20,806 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:09:20,859 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:09:20,945 DEBUG awx.main.dispatch task 03510177-70cf-4c85-a77e-6604afe34d24 is finished
2022-08-28 03:09:28,456 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2cfae08a-6370-411d-a3c8-ca9fa2a6d06c, queue=awx_private_queue)
[2022-08-28 03:09:28,456: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2cfae08a-6370-411d-a3c8-ca9fa2a6d06c, queue=awx_private_queue)
2022-08-28 03:09:28,470 DEBUG awx.main.dispatch delivered 2cfae08a-6370-411d-a3c8-ca9fa2a6d06c to worker[182] qsize 0
2022-08-28 03:09:28,473 DEBUG awx.main.dispatch task 2cfae08a-6370-411d-a3c8-ca9fa2a6d06c starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:09:28,491 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:09:28,495 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:08:58.507243+00:00
2022-08-28 03:09:28,513 DEBUG awx.main.dispatch task 2cfae08a-6370-411d-a3c8-ca9fa2a6d06c is finished
2022-08-28 03:09:40,781 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(48a58daa-bae2-4dfd-83fe-b5f2568bfa53, queue=awx_private_queue)
[2022-08-28 03:09:40,781: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(48a58daa-bae2-4dfd-83fe-b5f2568bfa53, queue=awx_private_queue)
2022-08-28 03:09:40,866 DEBUG awx.main.dispatch delivered 48a58daa-bae2-4dfd-83fe-b5f2568bfa53 to worker[183] qsize 0
2022-08-28 03:09:41,103 DEBUG awx.main.dispatch task 48a58daa-bae2-4dfd-83fe-b5f2568bfa53 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:09:41,105 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:09:41,140 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:09:41,253 DEBUG awx.main.dispatch task 48a58daa-bae2-4dfd-83fe-b5f2568bfa53 is finished
2022-08-28 03:09:56,369 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(ca6eaa1a-f461-4447-bfc0-10e2fd5eeaf0, queue=onap-awx-0)
[2022-08-28 03:09:56,369: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(ca6eaa1a-f461-4447-bfc0-10e2fd5eeaf0, queue=onap-awx-0)
2022-08-28 03:09:56,404 DEBUG awx.main.dispatch delivered ca6eaa1a-f461-4447-bfc0-10e2fd5eeaf0 to worker[183] qsize 0
2022-08-28 03:09:56,406 DEBUG awx.main.dispatch task ca6eaa1a-f461-4447-bfc0-10e2fd5eeaf0 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:09:56,408 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:09:56,439 DEBUG awx.main.dispatch task ca6eaa1a-f461-4447-bfc0-10e2fd5eeaf0 is finished
2022-08-28 03:09:57,401 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(0433b742-7b6f-491b-9516-5903a9d095a0, queue=onap-awx-0)
[2022-08-28 03:09:57,401: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(0433b742-7b6f-491b-9516-5903a9d095a0, queue=onap-awx-0)
2022-08-28 03:09:57,416 DEBUG awx.main.dispatch delivered 0433b742-7b6f-491b-9516-5903a9d095a0 to worker[181] qsize 0
2022-08-28 03:09:57,419 DEBUG awx.main.dispatch task 0433b742-7b6f-491b-9516-5903a9d095a0 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:09:57,436 DEBUG awx.main.dispatch task 0433b742-7b6f-491b-9516-5903a9d095a0 is finished
2022-08-28 03:09:58,455 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8ce20532-8973-432b-9efb-1b4de6e999c1, queue=awx_private_queue)
[2022-08-28 03:09:58,455: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8ce20532-8973-432b-9efb-1b4de6e999c1, queue=awx_private_queue)
2022-08-28 03:09:58,470 DEBUG awx.main.dispatch delivered 8ce20532-8973-432b-9efb-1b4de6e999c1 to worker[180] qsize 0
2022-08-28 03:09:58,473 DEBUG awx.main.dispatch task 8ce20532-8973-432b-9efb-1b4de6e999c1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:09:58,564 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:09:58,584 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:09:28.491369+00:00
2022-08-28 03:09:58,668 DEBUG awx.main.dispatch task 8ce20532-8973-432b-9efb-1b4de6e999c1 is finished
2022-08-28 03:10:00,781 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cfe650c0-67d2-4147-b298-a29f9d7e2034, queue=awx_private_queue)
[2022-08-28 03:10:00,781: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cfe650c0-67d2-4147-b298-a29f9d7e2034, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:10:00,796 DEBUG awx.main.dispatch delivered cfe650c0-67d2-4147-b298-a29f9d7e2034 to worker[180] qsize 0
2022-08-28 03:10:00,799 DEBUG awx.main.dispatch task cfe650c0-67d2-4147-b298-a29f9d7e2034 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:10:00,800 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:10:00,846 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:10:00,903 DEBUG awx.main.dispatch task cfe650c0-67d2-4147-b298-a29f9d7e2034 is finished
2022-08-28 03:10:20,790 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ad568ed1-e2a8-4eea-b595-da2927247391, queue=awx_private_queue)
[2022-08-28 03:10:20,790: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ad568ed1-e2a8-4eea-b595-da2927247391, queue=awx_private_queue)
2022-08-28 03:10:20,806 DEBUG awx.main.dispatch delivered ad568ed1-e2a8-4eea-b595-da2927247391 to worker[180] qsize 0
2022-08-28 03:10:20,809 DEBUG awx.main.dispatch task ad568ed1-e2a8-4eea-b595-da2927247391 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:10:20,810 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:10:20,847 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:10:20,911 DEBUG awx.main.dispatch task ad568ed1-e2a8-4eea-b595-da2927247391 is finished
2022-08-28 03:10:28,455 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8a3fec3a-dd54-449f-914e-1a0c8a344c24, queue=awx_private_queue)
[2022-08-28 03:10:28,455: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8a3fec3a-dd54-449f-914e-1a0c8a344c24, queue=awx_private_queue)
2022-08-28 03:10:28,472 DEBUG awx.main.dispatch delivered 8a3fec3a-dd54-449f-914e-1a0c8a344c24 to worker[183] qsize 0
2022-08-28 03:10:28,474 DEBUG awx.main.dispatch task 8a3fec3a-dd54-449f-914e-1a0c8a344c24 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:10:28,562 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:10:28,570 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:09:58.565138+00:00
2022-08-28 03:10:28,737 DEBUG awx.main.dispatch task 8a3fec3a-dd54-449f-914e-1a0c8a344c24 is finished
2022-08-28 03:10:40,793 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3ad1086d-b8d0-4077-bc16-f27980443285, queue=awx_private_queue)
[2022-08-28 03:10:40,793: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3ad1086d-b8d0-4077-bc16-f27980443285, queue=awx_private_queue)
2022-08-28 03:10:41,863 DEBUG awx.main.dispatch delivered 3ad1086d-b8d0-4077-bc16-f27980443285 to worker[182] qsize 0
2022-08-28 03:10:42,081 DEBUG awx.main.dispatch task 3ad1086d-b8d0-4077-bc16-f27980443285 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:10:42,083 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:10:43,114 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:10:43,119 DEBUG awx.main.dispatch task 3ad1086d-b8d0-4077-bc16-f27980443285 is finished
2022-08-28 03:10:56,371 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(08d4c855-5942-456f-adee-ad0f4f3cf214, queue=onap-awx-0)
[2022-08-28 03:10:56,371: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(08d4c855-5942-456f-adee-ad0f4f3cf214, queue=onap-awx-0)
2022-08-28 03:10:56,411 DEBUG awx.main.dispatch delivered 08d4c855-5942-456f-adee-ad0f4f3cf214 to worker[183] qsize 0
2022-08-28 03:10:56,413 DEBUG awx.main.dispatch task 08d4c855-5942-456f-adee-ad0f4f3cf214 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:10:56,415 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:10:56,561 DEBUG awx.main.dispatch task 08d4c855-5942-456f-adee-ad0f4f3cf214 is finished
2022-08-28 03:10:57,401 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c91530f5-a1c9-4c94-86fd-8f448b3e4346, queue=onap-awx-0)
[2022-08-28 03:10:57,401: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c91530f5-a1c9-4c94-86fd-8f448b3e4346, queue=onap-awx-0)
2022-08-28 03:10:57,423 DEBUG awx.main.dispatch delivered c91530f5-a1c9-4c94-86fd-8f448b3e4346 to worker[181] qsize 0
2022-08-28 03:10:57,425 DEBUG awx.main.dispatch task c91530f5-a1c9-4c94-86fd-8f448b3e4346 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:10:57,462 DEBUG awx.main.dispatch task c91530f5-a1c9-4c94-86fd-8f448b3e4346 is finished
2022-08-28 03:10:58,456 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5437eafb-d88f-4330-b3cb-d9063ea67e3e, queue=awx_private_queue)
[2022-08-28 03:10:58,456: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5437eafb-d88f-4330-b3cb-d9063ea67e3e, queue=awx_private_queue)
2022-08-28 03:10:58,472 DEBUG awx.main.dispatch delivered 5437eafb-d88f-4330-b3cb-d9063ea67e3e to worker[181] qsize 0
2022-08-28 03:10:58,474 DEBUG awx.main.dispatch task 5437eafb-d88f-4330-b3cb-d9063ea67e3e starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:10:58,498 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:10:58,504 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:10:28.562815+00:00
2022-08-28 03:10:58,535 DEBUG awx.main.dispatch task 5437eafb-d88f-4330-b3cb-d9063ea67e3e is finished
2022-08-28 03:11:00,793 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7c5205de-407a-4f08-8aea-b5954d264609, queue=awx_private_queue)
[2022-08-28 03:11:00,793: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7c5205de-407a-4f08-8aea-b5954d264609, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:11:00,809 DEBUG awx.main.dispatch delivered 7c5205de-407a-4f08-8aea-b5954d264609 to worker[180] qsize 0
2022-08-28 03:11:00,812 DEBUG awx.main.dispatch task 7c5205de-407a-4f08-8aea-b5954d264609 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:11:00,814 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:11:00,841 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:11:00,928 DEBUG awx.main.dispatch task 7c5205de-407a-4f08-8aea-b5954d264609 is finished
2022-08-28 03:11:20,802 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(46a982d6-6363-4af5-a9c1-d8fdc5f021b0, queue=awx_private_queue)
[2022-08-28 03:11:20,802: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(46a982d6-6363-4af5-a9c1-d8fdc5f021b0, queue=awx_private_queue)
2022-08-28 03:11:20,817 DEBUG awx.main.dispatch delivered 46a982d6-6363-4af5-a9c1-d8fdc5f021b0 to worker[182] qsize 0
2022-08-28 03:11:20,819 DEBUG awx.main.dispatch task 46a982d6-6363-4af5-a9c1-d8fdc5f021b0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:11:20,821 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:11:20,949 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:11:21,217 DEBUG awx.main.dispatch task 46a982d6-6363-4af5-a9c1-d8fdc5f021b0 is finished
2022-08-28 03:11:28,455 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(89beecb7-be55-414a-9243-0a4e4b289660, queue=awx_private_queue)
[2022-08-28 03:11:28,455: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(89beecb7-be55-414a-9243-0a4e4b289660, queue=awx_private_queue)
2022-08-28 03:11:28,472 DEBUG awx.main.dispatch delivered 89beecb7-be55-414a-9243-0a4e4b289660 to worker[181] qsize 0
2022-08-28 03:11:28,474 DEBUG awx.main.dispatch task 89beecb7-be55-414a-9243-0a4e4b289660 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:11:28,997 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:11:29,086 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:10:58.498801+00:00
2022-08-28 03:11:29,314 DEBUG awx.main.dispatch task 89beecb7-be55-414a-9243-0a4e4b289660 is finished
2022-08-28 03:11:40,807 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ce65e3a2-e960-42f0-a3a4-d1fde1799de9, queue=awx_private_queue)
[2022-08-28 03:11:40,807: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ce65e3a2-e960-42f0-a3a4-d1fde1799de9, queue=awx_private_queue)
2022-08-28 03:11:40,826 DEBUG awx.main.dispatch delivered ce65e3a2-e960-42f0-a3a4-d1fde1799de9 to worker[181] qsize 0
2022-08-28 03:11:40,829 DEBUG awx.main.dispatch task ce65e3a2-e960-42f0-a3a4-d1fde1799de9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:11:40,831 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:11:41,123 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:11:41,685 DEBUG awx.main.dispatch task ce65e3a2-e960-42f0-a3a4-d1fde1799de9 is finished
2022-08-28 03:11:56,261 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(730790cb-d9a3-45b7-a1f7-d2dc683161af, queue=onap-awx-0)
[2022-08-28 03:11:56,261: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(730790cb-d9a3-45b7-a1f7-d2dc683161af, queue=onap-awx-0)
2022-08-28 03:11:56,371 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a4df825d-b808-44c2-82ca-b67de2c0e625, queue=onap-awx-0)
[2022-08-28 03:11:56,371: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a4df825d-b808-44c2-82ca-b67de2c0e625, queue=onap-awx-0)
2022-08-28 03:11:56,289 DEBUG awx.main.dispatch delivered 730790cb-d9a3-45b7-a1f7-d2dc683161af to worker[180] qsize 0
2022-08-28 03:11:56,492 DEBUG awx.main.dispatch task 730790cb-d9a3-45b7-a1f7-d2dc683161af starting awx.main.tasks.awx_isolated_heartbeat(*[])
2022-08-28 03:11:56,495 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2022-08-28 03:11:56,705 DEBUG awx.main.dispatch delivered a4df825d-b808-44c2-82ca-b67de2c0e625 to worker[181] qsize 0
2022-08-28 03:11:56,707 DEBUG awx.main.dispatch task a4df825d-b808-44c2-82ca-b67de2c0e625 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:11:56,708 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:11:56,848 DEBUG awx.main.dispatch task 730790cb-d9a3-45b7-a1f7-d2dc683161af is finished
2022-08-28 03:11:57,104 DEBUG awx.main.dispatch task a4df825d-b808-44c2-82ca-b67de2c0e625 is finished
2022-08-28 03:11:57,400 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(34a8b05d-5c24-43d2-9805-4e6b754fb408, queue=onap-awx-0)
[2022-08-28 03:11:57,400: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(34a8b05d-5c24-43d2-9805-4e6b754fb408, queue=onap-awx-0)
2022-08-28 03:11:57,416 DEBUG awx.main.dispatch delivered 34a8b05d-5c24-43d2-9805-4e6b754fb408 to worker[182] qsize 0
2022-08-28 03:11:57,420 DEBUG awx.main.dispatch task 34a8b05d-5c24-43d2-9805-4e6b754fb408 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:11:57,474 DEBUG awx.main.dispatch task 34a8b05d-5c24-43d2-9805-4e6b754fb408 is finished
2022-08-28 03:11:58,456 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1bad8daa-9ba7-4aff-a66e-173cf938c307, queue=awx_private_queue)
[2022-08-28 03:11:58,456: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1bad8daa-9ba7-4aff-a66e-173cf938c307, queue=awx_private_queue)
2022-08-28 03:11:58,471 DEBUG awx.main.dispatch delivered 1bad8daa-9ba7-4aff-a66e-173cf938c307 to worker[181] qsize 0
2022-08-28 03:11:58,473 DEBUG awx.main.dispatch task 1bad8daa-9ba7-4aff-a66e-173cf938c307 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:11:58,587 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:11:58,599 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:11:28.997868+00:00
2022-08-28 03:11:58,878 DEBUG awx.main.dispatch task 1bad8daa-9ba7-4aff-a66e-173cf938c307 is finished
2022-08-28 03:12:00,806 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(65fdaace-9b14-4367-934c-57e50e2d9bca, queue=awx_private_queue)
[2022-08-28 03:12:00,806: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(65fdaace-9b14-4367-934c-57e50e2d9bca, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:12:00,823 DEBUG awx.main.dispatch delivered 65fdaace-9b14-4367-934c-57e50e2d9bca to worker[182] qsize 0
2022-08-28 03:12:00,825 DEBUG awx.main.dispatch task 65fdaace-9b14-4367-934c-57e50e2d9bca starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:12:00,827 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:12:00,955 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:12:01,332 DEBUG awx.main.dispatch task 65fdaace-9b14-4367-934c-57e50e2d9bca is finished
2022-08-28 03:12:20,820 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6712e2ae-b61b-4d32-9d97-22624bf4e3d3, queue=awx_private_queue)
[2022-08-28 03:12:20,820: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6712e2ae-b61b-4d32-9d97-22624bf4e3d3, queue=awx_private_queue)
2022-08-28 03:12:20,850 DEBUG awx.main.dispatch delivered 6712e2ae-b61b-4d32-9d97-22624bf4e3d3 to worker[182] qsize 0
2022-08-28 03:12:20,853 DEBUG awx.main.dispatch task 6712e2ae-b61b-4d32-9d97-22624bf4e3d3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:12:20,854 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:12:28,456 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4d0b9209-ccca-43f1-a87d-a99cea571b30, queue=awx_private_queue)
[2022-08-28 03:12:28,456: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4d0b9209-ccca-43f1-a87d-a99cea571b30, queue=awx_private_queue)
2022-08-28 03:12:28,472 DEBUG awx.main.dispatch delivered 4d0b9209-ccca-43f1-a87d-a99cea571b30 to worker[183] qsize 0
2022-08-28 03:12:28,476 DEBUG awx.main.dispatch task 4d0b9209-ccca-43f1-a87d-a99cea571b30 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:12:28,760 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:12:28,812 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:11:58.588229+00:00
2022-08-28 03:12:28,970 DEBUG awx.main.dispatch task 4d0b9209-ccca-43f1-a87d-a99cea571b30 is finished
2022-08-28 03:12:40,819 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0975e726-66f1-4f9d-8efe-600804448c84, queue=awx_private_queue)
[2022-08-28 03:12:40,819: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0975e726-66f1-4f9d-8efe-600804448c84, queue=awx_private_queue)
2022-08-28 03:12:40,836 DEBUG awx.main.dispatch delivered 0975e726-66f1-4f9d-8efe-600804448c84 to worker[180] qsize 0
2022-08-28 03:12:40,840 DEBUG awx.main.dispatch task 0975e726-66f1-4f9d-8efe-600804448c84 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:12:40,843 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:12:40,928 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:12:41,127 DEBUG awx.main.dispatch task 0975e726-66f1-4f9d-8efe-600804448c84 is finished
2022-08-28 03:12:56,374 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(445667ac-5995-4347-80b1-74654aeda4c5, queue=onap-awx-0)
[2022-08-28 03:12:56,374: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(445667ac-5995-4347-80b1-74654aeda4c5, queue=onap-awx-0)
2022-08-28 03:12:56,617 DEBUG awx.main.dispatch delivered 445667ac-5995-4347-80b1-74654aeda4c5 to worker[180] qsize 0
2022-08-28 03:12:56,832 DEBUG awx.main.dispatch task 445667ac-5995-4347-80b1-74654aeda4c5 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:12:56,833 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:12:57,095 DEBUG awx.main.dispatch task 445667ac-5995-4347-80b1-74654aeda4c5 is finished
2022-08-28 03:12:57,401 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(bda4857b-fcd1-4301-9652-ec6edfbf465e, queue=onap-awx-0)
[2022-08-28 03:12:57,401: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(bda4857b-fcd1-4301-9652-ec6edfbf465e, queue=onap-awx-0)
2022-08-28 03:12:57,430 DEBUG awx.main.dispatch delivered bda4857b-fcd1-4301-9652-ec6edfbf465e to worker[181] qsize 0
2022-08-28 03:12:57,432 DEBUG awx.main.dispatch task bda4857b-fcd1-4301-9652-ec6edfbf465e starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:12:57,706 DEBUG awx.main.dispatch task bda4857b-fcd1-4301-9652-ec6edfbf465e is finished
2022-08-28 03:12:58,456 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(07d8fc01-7bdf-4495-aa11-70779b5d4ddb, queue=awx_private_queue)
[2022-08-28 03:12:58,456: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(07d8fc01-7bdf-4495-aa11-70779b5d4ddb, queue=awx_private_queue)
2022-08-28 03:12:58,471 DEBUG awx.main.dispatch delivered 07d8fc01-7bdf-4495-aa11-70779b5d4ddb to worker[180] qsize 0
2022-08-28 03:12:58,473 DEBUG awx.main.dispatch task 07d8fc01-7bdf-4495-aa11-70779b5d4ddb starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:12:58,618 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:12:58,641 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:12:28.760747+00:00
2022-08-28 03:12:58,763 DEBUG awx.main.dispatch task 07d8fc01-7bdf-4495-aa11-70779b5d4ddb is finished
2022-08-28 03:13:00,819 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(00566ae4-40b4-461a-8a33-8a9b508a2ed0, queue=awx_private_queue)
[2022-08-28 03:13:00,819: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(00566ae4-40b4-461a-8a33-8a9b508a2ed0, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:13:00,834 DEBUG awx.main.dispatch delivered 00566ae4-40b4-461a-8a33-8a9b508a2ed0 to worker[180] qsize 0
2022-08-28 03:13:00,837 DEBUG awx.main.dispatch task 00566ae4-40b4-461a-8a33-8a9b508a2ed0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:13:00,838 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:13:00,977 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:13:01,145 DEBUG awx.main.dispatch task 00566ae4-40b4-461a-8a33-8a9b508a2ed0 is finished
2022-08-28 03:13:20,820 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3efb3086-459d-4606-bdd6-17d5ea4baaa6, queue=awx_private_queue)
[2022-08-28 03:13:20,820: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3efb3086-459d-4606-bdd6-17d5ea4baaa6, queue=awx_private_queue)
2022-08-28 03:13:20,859 DEBUG awx.main.dispatch delivered 3efb3086-459d-4606-bdd6-17d5ea4baaa6 to worker[181] qsize 0
2022-08-28 03:13:20,862 DEBUG awx.main.dispatch task 3efb3086-459d-4606-bdd6-17d5ea4baaa6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:13:20,864 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:13:20,930 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:13:21,017 DEBUG awx.main.dispatch task 3efb3086-459d-4606-bdd6-17d5ea4baaa6 is finished
2022-08-28 03:13:28,456 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bd721376-eedf-47d7-b362-b55a1cdddb2a, queue=awx_private_queue)
[2022-08-28 03:13:28,456: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bd721376-eedf-47d7-b362-b55a1cdddb2a, queue=awx_private_queue)
2022-08-28 03:13:28,472 DEBUG awx.main.dispatch delivered bd721376-eedf-47d7-b362-b55a1cdddb2a to worker[183] qsize 0
2022-08-28 03:13:28,474 DEBUG awx.main.dispatch task bd721376-eedf-47d7-b362-b55a1cdddb2a starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:13:28,529 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:13:28,550 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:12:58.619019+00:00
2022-08-28 03:13:30,352 DEBUG awx.main.dispatch task bd721376-eedf-47d7-b362-b55a1cdddb2a is finished
2022-08-28 03:13:40,822 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1918a4d1-de90-4932-a3dd-510ec52c6bf1, queue=awx_private_queue)
[2022-08-28 03:13:40,822: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1918a4d1-de90-4932-a3dd-510ec52c6bf1, queue=awx_private_queue)
2022-08-28 03:13:40,836 DEBUG awx.main.dispatch delivered 1918a4d1-de90-4932-a3dd-510ec52c6bf1 to worker[180] qsize 0
2022-08-28 03:13:40,839 DEBUG awx.main.dispatch task 1918a4d1-de90-4932-a3dd-510ec52c6bf1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:13:40,840 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:13:40,887 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:13:41,035 DEBUG awx.main.dispatch task 1918a4d1-de90-4932-a3dd-510ec52c6bf1 is finished
2022-08-28 03:13:56,378 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a393b56b-b3da-4d06-a6b9-bbfb0bd089c9, queue=onap-awx-0)
[2022-08-28 03:13:56,378: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a393b56b-b3da-4d06-a6b9-bbfb0bd089c9, queue=onap-awx-0)
2022-08-28 03:13:56,513 DEBUG awx.main.dispatch delivered a393b56b-b3da-4d06-a6b9-bbfb0bd089c9 to worker[180] qsize 0
2022-08-28 03:13:56,516 DEBUG awx.main.dispatch task a393b56b-b3da-4d06-a6b9-bbfb0bd089c9 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:13:56,518 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:13:56,677 DEBUG awx.main.dispatch task a393b56b-b3da-4d06-a6b9-bbfb0bd089c9 is finished
2022-08-28 03:13:57,401 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(83acb292-84f4-4d3b-a57e-b755351aa4d9, queue=onap-awx-0)
[2022-08-28 03:13:57,401: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(83acb292-84f4-4d3b-a57e-b755351aa4d9, queue=onap-awx-0)
2022-08-28 03:13:57,416 DEBUG awx.main.dispatch delivered 83acb292-84f4-4d3b-a57e-b755351aa4d9 to worker[181] qsize 0
2022-08-28 03:13:57,619 DEBUG awx.main.dispatch task 83acb292-84f4-4d3b-a57e-b755351aa4d9 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:13:57,753 DEBUG awx.main.dispatch task 83acb292-84f4-4d3b-a57e-b755351aa4d9 is finished
2022-08-28 03:13:58,456 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(330eecb9-1920-4026-96d5-edce299175f5, queue=awx_private_queue)
[2022-08-28 03:13:58,456: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(330eecb9-1920-4026-96d5-edce299175f5, queue=awx_private_queue)
2022-08-28 03:13:58,471 DEBUG awx.main.dispatch delivered 330eecb9-1920-4026-96d5-edce299175f5 to worker[181] qsize 0
2022-08-28 03:13:58,474 DEBUG awx.main.dispatch task 330eecb9-1920-4026-96d5-edce299175f5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:13:58,537 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:13:58,573 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:13:28.530118+00:00
2022-08-28 03:13:58,688 DEBUG awx.main.dispatch task 330eecb9-1920-4026-96d5-edce299175f5 is finished
2022-08-28 03:14:00,822 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(91df8536-ca70-462d-bb44-9110d36b22fe, queue=awx_private_queue)
[2022-08-28 03:14:00,822: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(91df8536-ca70-462d-bb44-9110d36b22fe, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:14:00,842 DEBUG awx.main.dispatch delivered 91df8536-ca70-462d-bb44-9110d36b22fe to worker[181] qsize 0
2022-08-28 03:14:00,845 DEBUG awx.main.dispatch task 91df8536-ca70-462d-bb44-9110d36b22fe starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:14:00,846 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:14:01,001 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:14:01,203 DEBUG awx.main.dispatch task 91df8536-ca70-462d-bb44-9110d36b22fe is finished
2022-08-28 03:14:20,822 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5080a1e6-69aa-45e1-a7e0-3bc5d291b13c, queue=awx_private_queue)
[2022-08-28 03:14:20,822: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5080a1e6-69aa-45e1-a7e0-3bc5d291b13c, queue=awx_private_queue)
2022-08-28 03:14:20,853 DEBUG awx.main.dispatch delivered 5080a1e6-69aa-45e1-a7e0-3bc5d291b13c to worker[183] qsize 0
2022-08-28 03:14:20,855 DEBUG awx.main.dispatch task 5080a1e6-69aa-45e1-a7e0-3bc5d291b13c starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:14:20,856 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:14:21,064 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:14:21,196 DEBUG awx.main.dispatch task 5080a1e6-69aa-45e1-a7e0-3bc5d291b13c is finished
2022-08-28 03:14:28,456 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3479e2c9-ab88-4831-a128-f64fa53ab352, queue=awx_private_queue)
[2022-08-28 03:14:28,456: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3479e2c9-ab88-4831-a128-f64fa53ab352, queue=awx_private_queue)
2022-08-28 03:14:28,483 DEBUG awx.main.dispatch delivered 3479e2c9-ab88-4831-a128-f64fa53ab352 to worker[183] qsize 0
2022-08-28 03:14:28,485 DEBUG awx.main.dispatch task 3479e2c9-ab88-4831-a128-f64fa53ab352 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:14:28,507 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:14:28,510 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:13:58.537582+00:00
2022-08-28 03:14:28,559 DEBUG awx.main.dispatch task 3479e2c9-ab88-4831-a128-f64fa53ab352 is finished
2022-08-28 03:14:30,754 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:14:30,758 DEBUG awx.main.dispatch task 6712e2ae-b61b-4d32-9d97-22624bf4e3d3 is finished
2022-08-28 03:14:40,822 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3cad084e-79f7-4776-a6cf-c82d43407990, queue=awx_private_queue)
[2022-08-28 03:14:40,822: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3cad084e-79f7-4776-a6cf-c82d43407990, queue=awx_private_queue)
2022-08-28 03:14:40,838 DEBUG awx.main.dispatch delivered 3cad084e-79f7-4776-a6cf-c82d43407990 to worker[183] qsize 0
2022-08-28 03:14:40,841 DEBUG awx.main.dispatch task 3cad084e-79f7-4776-a6cf-c82d43407990 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:14:40,842 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:14:41,121 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:14:41,543 DEBUG awx.main.dispatch task 3cad084e-79f7-4776-a6cf-c82d43407990 is finished
2022-08-28 03:14:56,383 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(9c6afb4d-0e63-4d5a-952a-9112e52dea11, queue=onap-awx-0)
[2022-08-28 03:14:56,383: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(9c6afb4d-0e63-4d5a-952a-9112e52dea11, queue=onap-awx-0)
2022-08-28 03:14:56,443 DEBUG awx.main.dispatch delivered 9c6afb4d-0e63-4d5a-952a-9112e52dea11 to worker[181] qsize 0
2022-08-28 03:14:56,446 DEBUG awx.main.dispatch task 9c6afb4d-0e63-4d5a-952a-9112e52dea11 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:14:56,449 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:14:56,599 DEBUG awx.main.dispatch task 9c6afb4d-0e63-4d5a-952a-9112e52dea11 is finished
2022-08-28 03:14:57,401 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8f7f2e13-ccbf-49aa-a2f2-9cd317636e75, queue=onap-awx-0)
[2022-08-28 03:14:57,401: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8f7f2e13-ccbf-49aa-a2f2-9cd317636e75, queue=onap-awx-0)
2022-08-28 03:14:57,448 DEBUG awx.main.dispatch delivered 8f7f2e13-ccbf-49aa-a2f2-9cd317636e75 to worker[180] qsize 0
2022-08-28 03:14:57,643 DEBUG awx.main.dispatch task 8f7f2e13-ccbf-49aa-a2f2-9cd317636e75 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:14:57,679 DEBUG awx.main.dispatch task 8f7f2e13-ccbf-49aa-a2f2-9cd317636e75 is finished
2022-08-28 03:14:58,456 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(346bfdb6-658d-4d42-9b74-467c8f17bf41, queue=awx_private_queue)
[2022-08-28 03:14:58,456: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(346bfdb6-658d-4d42-9b74-467c8f17bf41, queue=awx_private_queue)
2022-08-28 03:14:58,477 DEBUG awx.main.dispatch delivered 346bfdb6-658d-4d42-9b74-467c8f17bf41 to worker[183] qsize 0
2022-08-28 03:14:58,479 DEBUG awx.main.dispatch task 346bfdb6-658d-4d42-9b74-467c8f17bf41 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:14:58,510 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:14:58,515 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:14:28.507399+00:00
2022-08-28 03:14:58,633 DEBUG awx.main.dispatch task 346bfdb6-658d-4d42-9b74-467c8f17bf41 is finished
2022-08-28 03:15:00,822 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(dace5032-2ed3-4c30-8ee0-4d1d5cfbc06b, queue=awx_private_queue)
[2022-08-28 03:15:00,822: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(dace5032-2ed3-4c30-8ee0-4d1d5cfbc06b, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:15:00,838 DEBUG awx.main.dispatch delivered dace5032-2ed3-4c30-8ee0-4d1d5cfbc06b to worker[180] qsize 0
2022-08-28 03:15:00,840 DEBUG awx.main.dispatch task dace5032-2ed3-4c30-8ee0-4d1d5cfbc06b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:15:00,841 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:15:00,903 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:15:00,998 DEBUG awx.main.dispatch task dace5032-2ed3-4c30-8ee0-4d1d5cfbc06b is finished
2022-08-28 03:15:20,829 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6f801f4e-6f30-4094-bcc6-09c6f3cddc97, queue=awx_private_queue)
[2022-08-28 03:15:20,829: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6f801f4e-6f30-4094-bcc6-09c6f3cddc97, queue=awx_private_queue)
2022-08-28 03:15:20,845 DEBUG awx.main.dispatch delivered 6f801f4e-6f30-4094-bcc6-09c6f3cddc97 to worker[183] qsize 0
2022-08-28 03:15:20,847 DEBUG awx.main.dispatch task 6f801f4e-6f30-4094-bcc6-09c6f3cddc97 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:15:20,849 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:15:21,097 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:15:21,423 DEBUG awx.main.dispatch task 6f801f4e-6f30-4094-bcc6-09c6f3cddc97 is finished
2022-08-28 03:15:28,457 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(69239e9b-e369-4093-b1ec-51af940d0548, queue=awx_private_queue)
[2022-08-28 03:15:28,457: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(69239e9b-e369-4093-b1ec-51af940d0548, queue=awx_private_queue)
2022-08-28 03:15:28,477 DEBUG awx.main.dispatch delivered 69239e9b-e369-4093-b1ec-51af940d0548 to worker[183] qsize 0
2022-08-28 03:15:28,480 DEBUG awx.main.dispatch task 69239e9b-e369-4093-b1ec-51af940d0548 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:15:28,543 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:15:28,550 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:14:58.511229+00:00
2022-08-28 03:15:28,592 DEBUG awx.main.dispatch task 69239e9b-e369-4093-b1ec-51af940d0548 is finished
2022-08-28 03:15:40,833 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c0fab0af-ed2c-4a6e-9083-037ae1e2777c, queue=awx_private_queue)
[2022-08-28 03:15:40,833: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c0fab0af-ed2c-4a6e-9083-037ae1e2777c, queue=awx_private_queue)
2022-08-28 03:15:40,852 DEBUG awx.main.dispatch delivered c0fab0af-ed2c-4a6e-9083-037ae1e2777c to worker[181] qsize 0
2022-08-28 03:15:40,857 DEBUG awx.main.dispatch task c0fab0af-ed2c-4a6e-9083-037ae1e2777c starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:15:40,859 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:15:41,892 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:15:41,896 DEBUG awx.main.dispatch task c0fab0af-ed2c-4a6e-9083-037ae1e2777c is finished
2022-08-28 03:15:56,387 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(24039889-368d-4d29-9cbe-28ce1d105f0a, queue=onap-awx-0)
[2022-08-28 03:15:56,387: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(24039889-368d-4d29-9cbe-28ce1d105f0a, queue=onap-awx-0)
2022-08-28 03:15:56,498 DEBUG awx.main.dispatch delivered 24039889-368d-4d29-9cbe-28ce1d105f0a to worker[183] qsize 0
2022-08-28 03:15:56,500 DEBUG awx.main.dispatch task 24039889-368d-4d29-9cbe-28ce1d105f0a starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:15:56,504 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:15:57,401 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d6e2c179-3040-41e2-b08d-9ca441473a27, queue=onap-awx-0)
[2022-08-28 03:15:57,401: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d6e2c179-3040-41e2-b08d-9ca441473a27, queue=onap-awx-0)
2022-08-28 03:15:57,539 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:15:58,457 DEBUG awx.main.dispatch delivered d6e2c179-3040-41e2-b08d-9ca441473a27 to worker[181] qsize 0
2022-08-28 03:15:58,699 DEBUG awx.main.dispatch task d6e2c179-3040-41e2-b08d-9ca441473a27 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:15:58,566 DEBUG awx.main.dispatch task 24039889-368d-4d29-9cbe-28ce1d105f0a is finished
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
2022-08-28 03:15:58,457 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(178cd89f-a996-486b-a9e6-f6040b0cbc1c, queue=awx_private_queue)
[2022-08-28 03:15:58,457: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(178cd89f-a996-486b-a9e6-f6040b0cbc1c, queue=awx_private_queue)
2022-08-28 03:18:07,859 DEBUG awx.main.dispatch delivered 178cd89f-a996-486b-a9e6-f6040b0cbc1c to worker[180] qsize 0
2022-08-28 03:18:08,126 DEBUG awx.main.dispatch task 178cd89f-a996-486b-a9e6-f6040b0cbc1c starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:18:07,860 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c0656076-2e7c-4f9d-bee0-5436597de585, queue=awx_private_queue)
[2022-08-28 03:18:07,860: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c0656076-2e7c-4f9d-bee0-5436597de585, queue=awx_private_queue)
2022-08-28 03:18:08,918 DEBUG awx.main.dispatch delivered c0656076-2e7c-4f9d-bee0-5436597de585 to worker[183] qsize 0
2022-08-28 03:18:08,920 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f4f975fb-18fd-4191-acda-26f391572361, queue=onap-awx-0)
[2022-08-28 03:18:08,920: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f4f975fb-18fd-4191-acda-26f391572361, queue=onap-awx-0)
2022-08-28 03:18:08,920 DEBUG awx.main.dispatch task c0656076-2e7c-4f9d-bee0-5436597de585 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:18:08,923 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:18:08,935 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(623de56d-27a9-40f0-9a94-155e9aed2bd2, queue=onap-awx-0)
[2022-08-28 03:18:08,935: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(623de56d-27a9-40f0-9a94-155e9aed2bd2, queue=onap-awx-0)
2022-08-28 03:18:09,137 DEBUG awx.main.dispatch delivered f4f975fb-18fd-4191-acda-26f391572361 to worker[182] qsize 0
2022-08-28 03:18:09,139 DEBUG awx.main.dispatch task f4f975fb-18fd-4191-acda-26f391572361 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:18:09,142 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:18:09,146 WARNING awx.main.dispatch scaling up worker pid:192
2022-08-28 03:18:09,152 DEBUG awx.main.dispatch delivered 623de56d-27a9-40f0-9a94-155e9aed2bd2 to worker[192] qsize 0
2022-08-28 03:18:09,155 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:18:09,158 DEBUG awx.main.dispatch task 178cd89f-a996-486b-a9e6-f6040b0cbc1c is finished
2022-08-28 03:18:09,157 DEBUG awx.main.dispatch task 623de56d-27a9-40f0-9a94-155e9aed2bd2 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:18:09,890 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection timed out
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection timed out
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:18:09,893 DEBUG awx.main.dispatch task d6e2c179-3040-41e2-b08d-9ca441473a27 is finished
2022-08-28 03:18:09,955 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:18:09,958 DEBUG awx.main.dispatch task c0656076-2e7c-4f9d-bee0-5436597de585 is finished
2022-08-28 03:18:12,197 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:18:12,200 DEBUG awx.main.dispatch task f4f975fb-18fd-4191-acda-26f391572361 is finished
2022-08-28 03:18:12,195 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:18:12,205 DEBUG awx.main.dispatch task 623de56d-27a9-40f0-9a94-155e9aed2bd2 is finished
2022-08-28 03:18:27,880 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9dcf07de-2ccb-42ac-8203-730f18bc2695, queue=awx_private_queue)
[2022-08-28 03:18:27,880: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9dcf07de-2ccb-42ac-8203-730f18bc2695, queue=awx_private_queue)
2022-08-28 03:18:27,898 DEBUG awx.main.dispatch delivered 9dcf07de-2ccb-42ac-8203-730f18bc2695 to worker[182] qsize 0
2022-08-28 03:18:27,899 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ec3fe8db-4ace-4cb0-a292-f44c87cfe573, queue=awx_private_queue)
[2022-08-28 03:18:27,899: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ec3fe8db-4ace-4cb0-a292-f44c87cfe573, queue=awx_private_queue)
2022-08-28 03:18:27,900 DEBUG awx.main.dispatch task 9dcf07de-2ccb-42ac-8203-730f18bc2695 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:18:27,902 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:18:27,919 DEBUG awx.main.dispatch delivered ec3fe8db-4ace-4cb0-a292-f44c87cfe573 to worker[181] qsize 0
2022-08-28 03:18:27,921 DEBUG awx.main.dispatch task ec3fe8db-4ace-4cb0-a292-f44c87cfe573 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:18:28,930 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:18:28,933 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:18:28,933 DEBUG awx.main.dispatch task 9dcf07de-2ccb-42ac-8203-730f18bc2695 is finished
2022-08-28 03:18:28,935 DEBUG awx.main.dispatch task ec3fe8db-4ace-4cb0-a292-f44c87cfe573 is finished
2022-08-28 03:18:47,882 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fc9c97a9-7c24-40e1-8df7-8e1b1bd55bfc, queue=awx_private_queue)
[2022-08-28 03:18:47,882: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fc9c97a9-7c24-40e1-8df7-8e1b1bd55bfc, queue=awx_private_queue)
2022-08-28 03:18:47,898 DEBUG awx.main.dispatch delivered fc9c97a9-7c24-40e1-8df7-8e1b1bd55bfc to worker[182] qsize 0
2022-08-28 03:18:47,900 DEBUG awx.main.dispatch task fc9c97a9-7c24-40e1-8df7-8e1b1bd55bfc starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:18:47,901 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:18:48,931 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:18:48,936 DEBUG awx.main.dispatch task fc9c97a9-7c24-40e1-8df7-8e1b1bd55bfc is finished
2022-08-28 03:18:57,900 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1e08c896-df28-42ad-9770-c111969585cd, queue=awx_private_queue)
[2022-08-28 03:18:57,900: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1e08c896-df28-42ad-9770-c111969585cd, queue=awx_private_queue)
2022-08-28 03:18:57,916 DEBUG awx.main.dispatch delivered 1e08c896-df28-42ad-9770-c111969585cd to worker[183] qsize 0
2022-08-28 03:18:57,919 DEBUG awx.main.dispatch task 1e08c896-df28-42ad-9770-c111969585cd starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:18:58,947 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:18:58,950 DEBUG awx.main.dispatch task 1e08c896-df28-42ad-9770-c111969585cd is finished
RESULT 2
OKREADY
2022-08-28 03:19:07,882 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(45646ec7-5642-46e2-afcc-610be7731758, queue=awx_private_queue)
[2022-08-28 03:19:07,882: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(45646ec7-5642-46e2-afcc-610be7731758, queue=awx_private_queue)
2022-08-28 03:19:08,937 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(4ef6d3ca-4828-4008-bbfb-98dccf5398f2, queue=onap-awx-0)
[2022-08-28 03:19:08,937: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(4ef6d3ca-4828-4008-bbfb-98dccf5398f2, queue=onap-awx-0)
2022-08-28 03:19:08,934 DEBUG awx.main.dispatch delivered 45646ec7-5642-46e2-afcc-610be7731758 to worker[192] qsize 0
2022-08-28 03:19:09,982 WARNING awx.main.dispatch scaling down worker pid:180
2022-08-28 03:19:11,014 WARNING awx.main.dispatch worker exiting gracefully pid:180
2022-08-28 03:19:09,971 DEBUG awx.main.dispatch task 45646ec7-5642-46e2-afcc-610be7731758 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:19:13,038 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:19:14,055 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 123, in register
with advisory_lock('instance_registration_%s' % hostname):
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:19:16,100 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:19:17,126 DEBUG awx.main.dispatch task 45646ec7-5642-46e2-afcc-610be7731758 is finished
2022-08-28 03:19:09,983 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b0731eb8-2e99-4d8c-bd2d-675d89d45702, queue=onap-awx-0)
[2022-08-28 03:19:09,983: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b0731eb8-2e99-4d8c-bd2d-675d89d45702, queue=onap-awx-0)
2022-08-28 03:19:25,433 DEBUG awx.main.dispatch delivered b0731eb8-2e99-4d8c-bd2d-675d89d45702 to worker[192] qsize 0
2022-08-28 03:19:26,473 DEBUG awx.main.dispatch task b0731eb8-2e99-4d8c-bd2d-675d89d45702 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:19:28,515 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:19:29,539 DEBUG awx.main.dispatch task b0731eb8-2e99-4d8c-bd2d-675d89d45702 is finished
2022-08-28 03:19:27,900 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c2740f93-133c-4282-81b9-68b73d582c08, queue=awx_private_queue)
[2022-08-28 03:19:27,900: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c2740f93-133c-4282-81b9-68b73d582c08, queue=awx_private_queue)
2022-08-28 03:19:35,161 DEBUG awx.main.dispatch delivered c2740f93-133c-4282-81b9-68b73d582c08 to worker[192] qsize 0
2022-08-28 03:19:35,163 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3ce5036f-1c5e-425f-8484-957a2856a5e3, queue=awx_private_queue)
[2022-08-28 03:19:35,163: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3ce5036f-1c5e-425f-8484-957a2856a5e3, queue=awx_private_queue)
2022-08-28 03:19:36,201 DEBUG awx.main.dispatch task c2740f93-133c-4282-81b9-68b73d582c08 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:19:38,243 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:19:39,268 DEBUG awx.main.dispatch task c2740f93-133c-4282-81b9-68b73d582c08 is finished
2022-08-28 03:19:55,171 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a6e281e9-f498-4026-b90e-16d24a240e82, queue=awx_private_queue)
[2022-08-28 03:19:55,171: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a6e281e9-f498-4026-b90e-16d24a240e82, queue=awx_private_queue)
2022-08-28 03:19:57,900 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c0554f07-f450-40fe-8877-52b5dd9a04b9, queue=awx_private_queue)
[2022-08-28 03:19:57,900: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c0554f07-f450-40fe-8877-52b5dd9a04b9, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:20:08,937 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(bca0da2d-c929-4d5f-9c47-cdc4b8bcc129, queue=onap-awx-0)
[2022-08-28 03:20:08,937: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(bca0da2d-c929-4d5f-9c47-cdc4b8bcc129, queue=onap-awx-0)
2022-08-28 03:20:15,171 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(faa47786-b248-4221-9ca0-c7bbc1cdf87e, queue=awx_private_queue)
[2022-08-28 03:20:15,171: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(faa47786-b248-4221-9ca0-c7bbc1cdf87e, queue=awx_private_queue)
2022-08-28 03:20:15,191 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c99cdacc-0c15-4596-996a-b8866dc9f345, queue=onap-awx-0)
[2022-08-28 03:20:15,191: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c99cdacc-0c15-4596-996a-b8866dc9f345, queue=onap-awx-0)
2022-08-28 03:20:27,900 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2b05c169-32d0-4a81-8437-28f62da0a1de, queue=awx_private_queue)
[2022-08-28 03:20:27,900: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2b05c169-32d0-4a81-8437-28f62da0a1de, queue=awx_private_queue)
2022-08-28 03:20:35,171 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0aca56f2-31c3-41fe-bc91-4e449466b414, queue=awx_private_queue)
[2022-08-28 03:20:35,171: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0aca56f2-31c3-41fe-bc91-4e449466b414, queue=awx_private_queue)
2022-08-28 03:20:55,181 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0394a9dd-0b52-4d06-97a1-b43f3633152b, queue=awx_private_queue)
[2022-08-28 03:20:55,181: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0394a9dd-0b52-4d06-97a1-b43f3633152b, queue=awx_private_queue)
2022-08-28 03:20:57,900 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(571ccb66-a47a-4359-91ad-216c9bdb9b57, queue=awx_private_queue)
[2022-08-28 03:20:57,900: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(571ccb66-a47a-4359-91ad-216c9bdb9b57, queue=awx_private_queue)
RESULT 2
OKREADY
2022-08-28 03:21:08,938 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(cac4dd31-dac3-4571-8e0c-97987f0bceec, queue=onap-awx-0)
[2022-08-28 03:21:08,938: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(cac4dd31-dac3-4571-8e0c-97987f0bceec, queue=onap-awx-0)
2022-08-28 03:21:15,181 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(27c3664d-e0e2-449e-b858-19c5ebcd613c, queue=awx_private_queue)
[2022-08-28 03:21:15,181: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(27c3664d-e0e2-449e-b858-19c5ebcd613c, queue=awx_private_queue)
2022-08-28 03:21:15,203 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(008c0ebc-0387-4107-a027-7150342dca62, queue=onap-awx-0)
[2022-08-28 03:21:15,203: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(008c0ebc-0387-4107-a027-7150342dca62, queue=onap-awx-0)
2022-08-28 03:21:27,901 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(74e257db-6d5e-41ca-9291-3fd1e14e9281, queue=awx_private_queue)
[2022-08-28 03:21:27,901: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(74e257db-6d5e-41ca-9291-3fd1e14e9281, queue=awx_private_queue)
2022-08-28 03:21:35,182 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e8159158-82dc-4e9c-8b20-40e5fc9b0f74, queue=awx_private_queue)
[2022-08-28 03:21:35,182: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e8159158-82dc-4e9c-8b20-40e5fc9b0f74, queue=awx_private_queue)
2022-08-28 03:19:36,219 DEBUG awx.main.dispatch delivered 3ce5036f-1c5e-425f-8484-957a2856a5e3 to worker[181] qsize 0
2022-08-28 03:21:46,982 DEBUG awx.main.dispatch task 3ce5036f-1c5e-425f-8484-957a2856a5e3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:21:47,224 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:21:46,980 DEBUG awx.main.dispatch delivered a6e281e9-f498-4026-b90e-16d24a240e82 to worker[183] qsize 0
2022-08-28 03:21:47,265 DEBUG awx.main.dispatch task a6e281e9-f498-4026-b90e-16d24a240e82 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:21:47,268 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:21:47,266 DEBUG awx.main.dispatch delivered c0554f07-f450-40fe-8877-52b5dd9a04b9 to worker[192] qsize 0
2022-08-28 03:21:47,271 DEBUG awx.main.dispatch task c0554f07-f450-40fe-8877-52b5dd9a04b9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:21:47,279 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:21:47,332 DEBUG awx.main.dispatch delivered bca0da2d-c929-4d5f-9c47-cdc4b8bcc129 to worker[182] qsize 0
2022-08-28 03:21:47,336 DEBUG awx.main.dispatch task bca0da2d-c929-4d5f-9c47-cdc4b8bcc129 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:21:47,342 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:21:47,346 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:21:47,344 WARNING awx.main.dispatch scaling up worker pid:195
2022-08-28 03:21:47,355 DEBUG awx.main.dispatch delivered faa47786-b248-4221-9ca0-c7bbc1cdf87e to worker[195] qsize 0
2022-08-28 03:21:47,353 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:15:28.543350+00:00
2022-08-28 03:21:47,362 DEBUG awx.main.dispatch task faa47786-b248-4221-9ca0-c7bbc1cdf87e starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:21:47,367 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:21:47,370 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2022-08-28 03:21:47,370 DEBUG awx.main.dispatch task a6e281e9-f498-4026-b90e-16d24a240e82 is finished
2022-08-28 03:21:47,368 WARNING awx.main.dispatch scaling up worker pid:197
2022-08-28 03:21:47,375 DEBUG awx.main.dispatch delivered c99cdacc-0c15-4596-996a-b8866dc9f345 to worker[183] qsize 0
2022-08-28 03:21:47,379 DEBUG awx.main.dispatch task c99cdacc-0c15-4596-996a-b8866dc9f345 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:21:47,382 DEBUG awx.main.dispatch delivered 2b05c169-32d0-4a81-8437-28f62da0a1de to worker[197] qsize 0
2022-08-28 03:21:47,393 DEBUG awx.main.dispatch task 2b05c169-32d0-4a81-8437-28f62da0a1de starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:21:47,405 DEBUG awx.main.dispatch task c0554f07-f450-40fe-8877-52b5dd9a04b9 is finished
2022-08-28 03:21:47,401 WARNING awx.main.dispatch scaling up worker pid:199
2022-08-28 03:21:47,407 DEBUG awx.main.dispatch delivered 0aca56f2-31c3-41fe-bc91-4e449466b414 to worker[199] qsize 0
2022-08-28 03:21:47,415 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2022-08-28 03:21:47,413 DEBUG awx.main.dispatch delivered 0394a9dd-0b52-4d06-97a1-b43f3633152b to worker[192] qsize 0
2022-08-28 03:21:47,416 DEBUG awx.main.dispatch task faa47786-b248-4221-9ca0-c7bbc1cdf87e is finished
2022-08-28 03:21:47,417 DEBUG awx.main.dispatch task 0aca56f2-31c3-41fe-bc91-4e449466b414 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:21:47,419 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:21:47,417 DEBUG awx.main.dispatch task 0394a9dd-0b52-4d06-97a1-b43f3633152b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:21:47,420 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:21:47,420 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:21:47,421 WARNING awx.main.tasks Rejoining the cluster as instance onap-awx-0.
2022-08-28 03:21:47,420 DEBUG awx.main.dispatch task 3ce5036f-1c5e-425f-8484-957a2856a5e3 is finished
2022-08-28 03:21:47,421 DEBUG awx.main.dispatch task bca0da2d-c929-4d5f-9c47-cdc4b8bcc129 is finished
2022-08-28 03:21:47,424 DEBUG awx.main.dispatch task c99cdacc-0c15-4596-996a-b8866dc9f345 is finished
2022-08-28 03:21:47,427 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:21:47.346509+00:00
2022-08-28 03:21:47,425 WARNING awx.main.dispatch scaling up worker pid:202
2022-08-28 03:21:47,431 DEBUG awx.main.dispatch delivered 571ccb66-a47a-4359-91ad-216c9bdb9b57 to worker[202] qsize 0
2022-08-28 03:21:47,437 WARNING awx.main.dispatch scaling down worker pid:181
2022-08-28 03:21:47,437 DEBUG awx.main.dispatch task 571ccb66-a47a-4359-91ad-216c9bdb9b57 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:21:47,440 WARNING awx.main.dispatch scaling down worker pid:182
2022-08-28 03:21:47,440 WARNING awx.main.dispatch worker exiting gracefully pid:181
2022-08-28 03:21:47,441 WARNING awx.main.dispatch scaling down worker pid:183
2022-08-28 03:21:47,443 WARNING awx.main.dispatch worker exiting gracefully pid:183
2022-08-28 03:21:47,445 WARNING awx.main.dispatch worker exiting gracefully pid:182
2022-08-28 03:21:47,445 WARNING awx.main.dispatch scaling down worker pid:195
2022-08-28 03:21:47,449 WARNING awx.main.dispatch worker exiting gracefully pid:195
2022-08-28 03:21:47,471 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:21:47,488 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2022-08-28 03:21:47,489 DEBUG awx.main.dispatch task 571ccb66-a47a-4359-91ad-216c9bdb9b57 is finished
2022-08-28 03:21:47,496 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2022-08-28 03:21:47,497 DEBUG awx.main.dispatch task 0aca56f2-31c3-41fe-bc91-4e449466b414 is finished
2022-08-28 03:21:47,509 DEBUG awx.main.dispatch task 2b05c169-32d0-4a81-8437-28f62da0a1de is finished
2022-08-28 03:21:47,574 DEBUG awx.main.dispatch delivered cac4dd31-dac3-4571-8e0c-97987f0bceec to worker[199] qsize 0
2022-08-28 03:21:47,577 DEBUG awx.main.dispatch task cac4dd31-dac3-4571-8e0c-97987f0bceec starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:21:47,577 DEBUG awx.main.dispatch delivered 27c3664d-e0e2-449e-b858-19c5ebcd613c to worker[197] qsize 0
2022-08-28 03:21:47,578 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:21:47,579 DEBUG awx.main.dispatch delivered 008c0ebc-0387-4107-a027-7150342dca62 to worker[202] qsize 0
2022-08-28 03:21:47,581 DEBUG awx.main.dispatch task 008c0ebc-0387-4107-a027-7150342dca62 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:21:47,582 DEBUG awx.main.dispatch task 27c3664d-e0e2-449e-b858-19c5ebcd613c starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:21:47,591 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:21:47,589 WARNING awx.main.dispatch scaling up worker pid:207
2022-08-28 03:21:47,596 DEBUG awx.main.dispatch delivered 74e257db-6d5e-41ca-9291-3fd1e14e9281 to worker[207] qsize 0
2022-08-28 03:21:47,602 DEBUG awx.main.dispatch task 74e257db-6d5e-41ca-9291-3fd1e14e9281 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:21:47,611 DEBUG awx.main.dispatch task 0394a9dd-0b52-4d06-97a1-b43f3633152b is finished
2022-08-28 03:21:47,610 WARNING awx.main.dispatch scaling up worker pid:209
2022-08-28 03:21:47,617 DEBUG awx.main.dispatch delivered e8159158-82dc-4e9c-8b20-40e5fc9b0f74 to worker[209] qsize 0
2022-08-28 03:21:47,625 DEBUG awx.main.dispatch task e8159158-82dc-4e9c-8b20-40e5fc9b0f74 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:21:47,627 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:21:47,635 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:21:47,643 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:21:47.420065+00:00
2022-08-28 03:21:47,669 DEBUG awx.main.dispatch task 008c0ebc-0387-4107-a027-7150342dca62 is finished
2022-08-28 03:21:47,670 DEBUG awx.main.dispatch task cac4dd31-dac3-4571-8e0c-97987f0bceec is finished
2022-08-28 03:21:47,687 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:21:47,687 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2022-08-28 03:21:47,687 DEBUG awx.main.dispatch task 27c3664d-e0e2-449e-b858-19c5ebcd613c is finished
2022-08-28 03:21:47,727 DEBUG awx.main.dispatch task 74e257db-6d5e-41ca-9291-3fd1e14e9281 is finished
2022-08-28 03:21:47,822 DEBUG awx.main.dispatch task e8159158-82dc-4e9c-8b20-40e5fc9b0f74 is finished
2022-08-28 03:21:55,192 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c3a4a3cc-b0ec-49b5-a850-752fa8ddbef4, queue=awx_private_queue)
[2022-08-28 03:21:55,192: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c3a4a3cc-b0ec-49b5-a850-752fa8ddbef4, queue=awx_private_queue)
2022-08-28 03:21:55,209 DEBUG awx.main.dispatch delivered c3a4a3cc-b0ec-49b5-a850-752fa8ddbef4 to worker[197] qsize 0
2022-08-28 03:21:55,211 DEBUG awx.main.dispatch task c3a4a3cc-b0ec-49b5-a850-752fa8ddbef4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:21:55,213 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:21:55,276 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:21:55,439 DEBUG awx.main.dispatch task c3a4a3cc-b0ec-49b5-a850-752fa8ddbef4 is finished
2022-08-28 03:21:56,257 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(a564f4e2-e31f-40d0-b2dd-00ca365bbe53, queue=onap-awx-0)
[2022-08-28 03:21:56,257: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(a564f4e2-e31f-40d0-b2dd-00ca365bbe53, queue=onap-awx-0)
2022-08-28 03:21:56,275 DEBUG awx.main.dispatch delivered a564f4e2-e31f-40d0-b2dd-00ca365bbe53 to worker[192] qsize 0
2022-08-28 03:21:56,277 DEBUG awx.main.dispatch task a564f4e2-e31f-40d0-b2dd-00ca365bbe53 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2022-08-28 03:21:56,278 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2022-08-28 03:21:56,374 DEBUG awx.main.dispatch task a564f4e2-e31f-40d0-b2dd-00ca365bbe53 is finished
2022-08-28 03:21:57,901 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(12edadd2-d05e-492c-97ce-144d2ff5cc3a, queue=awx_private_queue)
[2022-08-28 03:21:57,901: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(12edadd2-d05e-492c-97ce-144d2ff5cc3a, queue=awx_private_queue)
2022-08-28 03:21:57,917 DEBUG awx.main.dispatch delivered 12edadd2-d05e-492c-97ce-144d2ff5cc3a to worker[199] qsize 0
2022-08-28 03:21:57,920 DEBUG awx.main.dispatch task 12edadd2-d05e-492c-97ce-144d2ff5cc3a starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:21:57,936 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:21:57,940 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:21:47.636305+00:00
2022-08-28 03:21:58,021 DEBUG awx.main.dispatch task 12edadd2-d05e-492c-97ce-144d2ff5cc3a is finished
RESULT 2
OKREADY
2022-08-28 03:22:08,938 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f876afbd-ef31-4641-a8c5-7982e993e708, queue=onap-awx-0)
[2022-08-28 03:22:08,938: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f876afbd-ef31-4641-a8c5-7982e993e708, queue=onap-awx-0)
2022-08-28 03:22:08,954 WARNING awx.main.dispatch scaling down worker pid:192
2022-08-28 03:22:08,956 WARNING awx.main.dispatch scaling down worker pid:197
2022-08-28 03:22:08,957 WARNING awx.main.dispatch worker exiting gracefully pid:192
2022-08-28 03:22:08,958 WARNING awx.main.dispatch worker exiting gracefully pid:197
2022-08-28 03:22:09,009 DEBUG awx.main.dispatch delivered f876afbd-ef31-4641-a8c5-7982e993e708 to worker[207] qsize 0
2022-08-28 03:22:09,011 DEBUG awx.main.dispatch task f876afbd-ef31-4641-a8c5-7982e993e708 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:22:09,013 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:22:09,048 DEBUG awx.main.dispatch task f876afbd-ef31-4641-a8c5-7982e993e708 is finished
2022-08-28 03:22:15,192 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(47cc9f4e-e6b5-4877-afb3-af12d2f220cd, queue=awx_private_queue)
[2022-08-28 03:22:15,192: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(47cc9f4e-e6b5-4877-afb3-af12d2f220cd, queue=awx_private_queue)
2022-08-28 03:22:15,209 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d5213474-3377-4af2-91a6-03008c51a0b8, queue=onap-awx-0)
[2022-08-28 03:22:15,209: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d5213474-3377-4af2-91a6-03008c51a0b8, queue=onap-awx-0)
2022-08-28 03:22:15,209 DEBUG awx.main.dispatch delivered 47cc9f4e-e6b5-4877-afb3-af12d2f220cd to worker[209] qsize 0
2022-08-28 03:22:15,212 DEBUG awx.main.dispatch task 47cc9f4e-e6b5-4877-afb3-af12d2f220cd starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:22:15,215 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:22:15,225 DEBUG awx.main.dispatch delivered d5213474-3377-4af2-91a6-03008c51a0b8 to worker[202] qsize 0
2022-08-28 03:22:15,227 DEBUG awx.main.dispatch task d5213474-3377-4af2-91a6-03008c51a0b8 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:22:15,291 DEBUG awx.main.dispatch task d5213474-3377-4af2-91a6-03008c51a0b8 is finished
2022-08-28 03:22:15,331 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:22:15,487 DEBUG awx.main.dispatch task 47cc9f4e-e6b5-4877-afb3-af12d2f220cd is finished
2022-08-28 03:22:27,902 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(44812188-62f1-49ec-b70b-da16ae1f2985, queue=awx_private_queue)
[2022-08-28 03:22:27,902: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(44812188-62f1-49ec-b70b-da16ae1f2985, queue=awx_private_queue)
2022-08-28 03:22:27,919 DEBUG awx.main.dispatch delivered 44812188-62f1-49ec-b70b-da16ae1f2985 to worker[202] qsize 0
2022-08-28 03:22:27,922 DEBUG awx.main.dispatch task 44812188-62f1-49ec-b70b-da16ae1f2985 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:22:27,961 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:22:27,969 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:21:57.936659+00:00
2022-08-28 03:22:28,024 DEBUG awx.main.dispatch task 44812188-62f1-49ec-b70b-da16ae1f2985 is finished
2022-08-28 03:22:35,192 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0884a5fd-4c13-44d8-9f06-210a38fff455, queue=awx_private_queue)
[2022-08-28 03:22:35,192: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0884a5fd-4c13-44d8-9f06-210a38fff455, queue=awx_private_queue)
2022-08-28 03:22:35,208 DEBUG awx.main.dispatch delivered 0884a5fd-4c13-44d8-9f06-210a38fff455 to worker[209] qsize 0
2022-08-28 03:22:35,211 DEBUG awx.main.dispatch task 0884a5fd-4c13-44d8-9f06-210a38fff455 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:22:35,212 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:22:35,236 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:22:35,272 DEBUG awx.main.dispatch task 0884a5fd-4c13-44d8-9f06-210a38fff455 is finished
2022-08-28 03:22:55,193 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bdf3db09-83b3-41a1-942a-9f84edf20c99, queue=awx_private_queue)
[2022-08-28 03:22:55,193: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bdf3db09-83b3-41a1-942a-9f84edf20c99, queue=awx_private_queue)
2022-08-28 03:22:55,306 DEBUG awx.main.dispatch delivered bdf3db09-83b3-41a1-942a-9f84edf20c99 to worker[207] qsize 0
2022-08-28 03:22:55,549 DEBUG awx.main.dispatch task bdf3db09-83b3-41a1-942a-9f84edf20c99 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:22:55,551 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:22:56,070 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:22:56,222 DEBUG awx.main.dispatch task bdf3db09-83b3-41a1-942a-9f84edf20c99 is finished
2022-08-28 03:22:57,902 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a708b956-c7b0-4d1c-bf76-9537f526c4dd, queue=awx_private_queue)
[2022-08-28 03:22:57,902: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a708b956-c7b0-4d1c-bf76-9537f526c4dd, queue=awx_private_queue)
2022-08-28 03:22:57,920 DEBUG awx.main.dispatch delivered a708b956-c7b0-4d1c-bf76-9537f526c4dd to worker[207] qsize 0
2022-08-28 03:22:57,923 DEBUG awx.main.dispatch task a708b956-c7b0-4d1c-bf76-9537f526c4dd starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:22:58,074 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:22:58,093 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:22:27.961790+00:00
2022-08-28 03:22:58,230 DEBUG awx.main.dispatch task a708b956-c7b0-4d1c-bf76-9537f526c4dd is finished
RESULT 2
OKREADY
2022-08-28 03:23:08,941 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(fd4ad9ed-38bb-4a80-ac2b-7d6561d52c44, queue=onap-awx-0)
[2022-08-28 03:23:08,941: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(fd4ad9ed-38bb-4a80-ac2b-7d6561d52c44, queue=onap-awx-0)
2022-08-28 03:23:09,175 DEBUG awx.main.dispatch delivered fd4ad9ed-38bb-4a80-ac2b-7d6561d52c44 to worker[207] qsize 0
2022-08-28 03:23:09,177 DEBUG awx.main.dispatch task fd4ad9ed-38bb-4a80-ac2b-7d6561d52c44 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:23:09,179 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:23:09,464 DEBUG awx.main.dispatch task fd4ad9ed-38bb-4a80-ac2b-7d6561d52c44 is finished
2022-08-28 03:23:15,209 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(97820213-11a7-4f4e-b6ed-d205d770c993, queue=onap-awx-0)
[2022-08-28 03:23:15,209: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(97820213-11a7-4f4e-b6ed-d205d770c993, queue=onap-awx-0)
2022-08-28 03:23:15,225 DEBUG awx.main.dispatch delivered 97820213-11a7-4f4e-b6ed-d205d770c993 to worker[199] qsize 0
2022-08-28 03:23:15,226 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d9aae39d-fc1b-4c25-95a8-5c3a283e46b3, queue=awx_private_queue)
[2022-08-28 03:23:15,226: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d9aae39d-fc1b-4c25-95a8-5c3a283e46b3, queue=awx_private_queue)
2022-08-28 03:23:15,228 DEBUG awx.main.dispatch task 97820213-11a7-4f4e-b6ed-d205d770c993 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:23:15,245 DEBUG awx.main.dispatch delivered d9aae39d-fc1b-4c25-95a8-5c3a283e46b3 to worker[202] qsize 0
2022-08-28 03:23:15,249 DEBUG awx.main.dispatch task d9aae39d-fc1b-4c25-95a8-5c3a283e46b3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:23:15,251 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:23:16,258 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:23:16,259 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:23:16,262 DEBUG awx.main.dispatch task 97820213-11a7-4f4e-b6ed-d205d770c993 is finished
2022-08-28 03:23:16,262 DEBUG awx.main.dispatch task d9aae39d-fc1b-4c25-95a8-5c3a283e46b3 is finished
2022-08-28 03:23:27,901 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5c2b1a0e-4032-469f-9767-326cf8e416e4, queue=awx_private_queue)
[2022-08-28 03:23:27,901: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5c2b1a0e-4032-469f-9767-326cf8e416e4, queue=awx_private_queue)
2022-08-28 03:23:27,918 DEBUG awx.main.dispatch delivered 5c2b1a0e-4032-469f-9767-326cf8e416e4 to worker[207] qsize 0
2022-08-28 03:23:27,920 DEBUG awx.main.dispatch task 5c2b1a0e-4032-469f-9767-326cf8e416e4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:23:28,967 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.233.12.116) and accepting
TCP/IP connections on port 5432?
2022-08-28 03:23:28,971 DEBUG awx.main.dispatch task 5c2b1a0e-4032-469f-9767-326cf8e416e4 is finished
2022-08-28 03:23:35,226 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a1c3374f-fbbb-4f9b-a12e-822cdf050aa0, queue=awx_private_queue)
[2022-08-28 03:23:35,226: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a1c3374f-fbbb-4f9b-a12e-822cdf050aa0, queue=awx_private_queue)
2022-08-28 03:23:35,241 DEBUG awx.main.dispatch delivered a1c3374f-fbbb-4f9b-a12e-822cdf050aa0 to worker[202] qsize 0
2022-08-28 03:23:35,243 DEBUG awx.main.dispatch task a1c3374f-fbbb-4f9b-a12e-822cdf050aa0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:23:35,245 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:23:35,447 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:23:35,632 DEBUG awx.main.dispatch task a1c3374f-fbbb-4f9b-a12e-822cdf050aa0 is finished
2022-08-28 03:23:55,234 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8c4b1c0e-2ccd-4460-a133-2d4e88d2f06e, queue=awx_private_queue)
[2022-08-28 03:23:55,234: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8c4b1c0e-2ccd-4460-a133-2d4e88d2f06e, queue=awx_private_queue)
2022-08-28 03:23:55,250 DEBUG awx.main.dispatch delivered 8c4b1c0e-2ccd-4460-a133-2d4e88d2f06e to worker[207] qsize 0
2022-08-28 03:23:55,252 DEBUG awx.main.dispatch task 8c4b1c0e-2ccd-4460-a133-2d4e88d2f06e starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:23:55,253 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:23:55,293 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:23:55,339 DEBUG awx.main.dispatch task 8c4b1c0e-2ccd-4460-a133-2d4e88d2f06e is finished
2022-08-28 03:23:57,901 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0f467e2e-af50-4370-86a1-68a804d6f97e, queue=awx_private_queue)
[2022-08-28 03:23:57,901: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0f467e2e-af50-4370-86a1-68a804d6f97e, queue=awx_private_queue)
2022-08-28 03:23:57,964 DEBUG awx.main.dispatch delivered 0f467e2e-af50-4370-86a1-68a804d6f97e to worker[209] qsize 0
2022-08-28 03:23:58,148 DEBUG awx.main.dispatch task 0f467e2e-af50-4370-86a1-68a804d6f97e starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:23:58,192 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:23:58,198 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:22:58.075254+00:00
2022-08-28 03:23:58,229 DEBUG awx.main.dispatch task 0f467e2e-af50-4370-86a1-68a804d6f97e is finished
RESULT 2
OKREADY
2022-08-28 03:24:08,943 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(53d2d692-29ff-4d33-9a6b-bd73e6285690, queue=onap-awx-0)
[2022-08-28 03:24:08,943: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(53d2d692-29ff-4d33-9a6b-bd73e6285690, queue=onap-awx-0)
2022-08-28 03:24:08,981 DEBUG awx.main.dispatch delivered 53d2d692-29ff-4d33-9a6b-bd73e6285690 to worker[207] qsize 0
2022-08-28 03:24:08,984 DEBUG awx.main.dispatch task 53d2d692-29ff-4d33-9a6b-bd73e6285690 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:24:08,986 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:24:09,023 DEBUG awx.main.dispatch task 53d2d692-29ff-4d33-9a6b-bd73e6285690 is finished
2022-08-28 03:24:15,209 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(80b33298-f611-4aa7-8041-4662a182da24, queue=onap-awx-0)
[2022-08-28 03:24:15,209: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(80b33298-f611-4aa7-8041-4662a182da24, queue=onap-awx-0)
2022-08-28 03:24:15,224 DEBUG awx.main.dispatch delivered 80b33298-f611-4aa7-8041-4662a182da24 to worker[209] qsize 0
2022-08-28 03:24:15,226 DEBUG awx.main.dispatch task 80b33298-f611-4aa7-8041-4662a182da24 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:24:15,234 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(28983cf4-fc3a-4c06-a5fe-63e3e07bd79a, queue=awx_private_queue)
[2022-08-28 03:24:15,234: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(28983cf4-fc3a-4c06-a5fe-63e3e07bd79a, queue=awx_private_queue)
2022-08-28 03:24:15,248 DEBUG awx.main.dispatch task 80b33298-f611-4aa7-8041-4662a182da24 is finished
2022-08-28 03:24:15,249 DEBUG awx.main.dispatch delivered 28983cf4-fc3a-4c06-a5fe-63e3e07bd79a to worker[202] qsize 0
2022-08-28 03:24:15,251 DEBUG awx.main.dispatch task 28983cf4-fc3a-4c06-a5fe-63e3e07bd79a starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:24:15,253 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:24:15,299 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:24:15,343 DEBUG awx.main.dispatch task 28983cf4-fc3a-4c06-a5fe-63e3e07bd79a is finished
2022-08-28 03:24:27,918 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c5c8ef6b-3005-40b7-a411-2c13474abe8a, queue=awx_private_queue)
[2022-08-28 03:24:27,918: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c5c8ef6b-3005-40b7-a411-2c13474abe8a, queue=awx_private_queue)
2022-08-28 03:24:27,933 DEBUG awx.main.dispatch delivered c5c8ef6b-3005-40b7-a411-2c13474abe8a to worker[209] qsize 0
2022-08-28 03:24:27,936 DEBUG awx.main.dispatch task c5c8ef6b-3005-40b7-a411-2c13474abe8a starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:24:27,948 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:24:27,951 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:23:58.193209+00:00
2022-08-28 03:24:27,980 DEBUG awx.main.dispatch task c5c8ef6b-3005-40b7-a411-2c13474abe8a is finished
2022-08-28 03:24:35,234 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(89e17030-24b2-4359-897f-4f3830387a62, queue=awx_private_queue)
[2022-08-28 03:24:35,234: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(89e17030-24b2-4359-897f-4f3830387a62, queue=awx_private_queue)
2022-08-28 03:24:35,250 DEBUG awx.main.dispatch delivered 89e17030-24b2-4359-897f-4f3830387a62 to worker[199] qsize 0
2022-08-28 03:24:35,252 DEBUG awx.main.dispatch task 89e17030-24b2-4359-897f-4f3830387a62 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:24:35,255 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:24:35,309 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:24:35,525 DEBUG awx.main.dispatch task 89e17030-24b2-4359-897f-4f3830387a62 is finished
2022-08-28 03:24:55,243 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a5672e51-ba70-4906-84d6-1b05bf99e9d4, queue=awx_private_queue)
[2022-08-28 03:24:55,243: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a5672e51-ba70-4906-84d6-1b05bf99e9d4, queue=awx_private_queue)
2022-08-28 03:24:55,258 DEBUG awx.main.dispatch delivered a5672e51-ba70-4906-84d6-1b05bf99e9d4 to worker[209] qsize 0
2022-08-28 03:24:55,261 DEBUG awx.main.dispatch task a5672e51-ba70-4906-84d6-1b05bf99e9d4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:24:55,262 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:24:55,322 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:24:55,455 DEBUG awx.main.dispatch task a5672e51-ba70-4906-84d6-1b05bf99e9d4 is finished
2022-08-28 03:24:57,918 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bc5a010b-3148-43d6-815c-047e93d78cfd, queue=awx_private_queue)
[2022-08-28 03:24:57,918: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bc5a010b-3148-43d6-815c-047e93d78cfd, queue=awx_private_queue)
2022-08-28 03:24:57,953 DEBUG awx.main.dispatch delivered bc5a010b-3148-43d6-815c-047e93d78cfd to worker[199] qsize 0
2022-08-28 03:24:57,955 DEBUG awx.main.dispatch task bc5a010b-3148-43d6-815c-047e93d78cfd starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:24:58,065 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:24:58,080 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:24:27.948406+00:00
2022-08-28 03:24:58,145 DEBUG awx.main.dispatch task bc5a010b-3148-43d6-815c-047e93d78cfd is finished
RESULT 2
OKREADY
2022-08-28 03:25:08,944 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(9677a712-5bb7-4ed5-9176-e8471e05c859, queue=onap-awx-0)
[2022-08-28 03:25:08,944: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(9677a712-5bb7-4ed5-9176-e8471e05c859, queue=onap-awx-0)
2022-08-28 03:25:09,204 DEBUG awx.main.dispatch delivered 9677a712-5bb7-4ed5-9176-e8471e05c859 to worker[209] qsize 0
2022-08-28 03:25:09,206 DEBUG awx.main.dispatch task 9677a712-5bb7-4ed5-9176-e8471e05c859 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:25:09,208 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:25:09,335 DEBUG awx.main.dispatch task 9677a712-5bb7-4ed5-9176-e8471e05c859 is finished
2022-08-28 03:25:15,211 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(7a894d47-2faa-4045-afc4-569566e242d2, queue=onap-awx-0)
[2022-08-28 03:25:15,211: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(7a894d47-2faa-4045-afc4-569566e242d2, queue=onap-awx-0)
2022-08-28 03:25:15,228 DEBUG awx.main.dispatch delivered 7a894d47-2faa-4045-afc4-569566e242d2 to worker[207] qsize 0
2022-08-28 03:25:15,232 DEBUG awx.main.dispatch task 7a894d47-2faa-4045-afc4-569566e242d2 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:25:15,244 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2f612e43-0886-4dbc-a633-c31b69578d1d, queue=awx_private_queue)
[2022-08-28 03:25:15,244: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2f612e43-0886-4dbc-a633-c31b69578d1d, queue=awx_private_queue)
2022-08-28 03:25:15,252 DEBUG awx.main.dispatch task 7a894d47-2faa-4045-afc4-569566e242d2 is finished
2022-08-28 03:25:15,264 DEBUG awx.main.dispatch delivered 2f612e43-0886-4dbc-a633-c31b69578d1d to worker[207] qsize 0
2022-08-28 03:25:15,267 DEBUG awx.main.dispatch task 2f612e43-0886-4dbc-a633-c31b69578d1d starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:25:15,269 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:25:15,291 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:25:15,334 DEBUG awx.main.dispatch task 2f612e43-0886-4dbc-a633-c31b69578d1d is finished
2022-08-28 03:25:27,919 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8ea47670-1d2f-4ce9-b2e7-532b80322e82, queue=awx_private_queue)
[2022-08-28 03:25:27,919: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8ea47670-1d2f-4ce9-b2e7-532b80322e82, queue=awx_private_queue)
2022-08-28 03:25:27,938 DEBUG awx.main.dispatch delivered 8ea47670-1d2f-4ce9-b2e7-532b80322e82 to worker[199] qsize 0
2022-08-28 03:25:27,942 DEBUG awx.main.dispatch task 8ea47670-1d2f-4ce9-b2e7-532b80322e82 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:25:27,956 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:25:27,962 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:24:58.066198+00:00
2022-08-28 03:25:27,979 DEBUG awx.main.dispatch task 8ea47670-1d2f-4ce9-b2e7-532b80322e82 is finished
2022-08-28 03:25:35,244 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(73149f1b-6abb-46c8-8bd6-f8ba53024600, queue=awx_private_queue)
[2022-08-28 03:25:35,244: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(73149f1b-6abb-46c8-8bd6-f8ba53024600, queue=awx_private_queue)
2022-08-28 03:25:35,261 DEBUG awx.main.dispatch delivered 73149f1b-6abb-46c8-8bd6-f8ba53024600 to worker[209] qsize 0
2022-08-28 03:25:35,265 DEBUG awx.main.dispatch task 73149f1b-6abb-46c8-8bd6-f8ba53024600 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:25:35,267 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:25:35,300 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:25:35,342 DEBUG awx.main.dispatch task 73149f1b-6abb-46c8-8bd6-f8ba53024600 is finished
2022-08-28 03:25:55,254 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4585f453-94c4-477b-ad78-b3b0f12379d3, queue=awx_private_queue)
[2022-08-28 03:25:55,254: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4585f453-94c4-477b-ad78-b3b0f12379d3, queue=awx_private_queue)
2022-08-28 03:25:55,270 DEBUG awx.main.dispatch delivered 4585f453-94c4-477b-ad78-b3b0f12379d3 to worker[202] qsize 0
2022-08-28 03:25:55,273 DEBUG awx.main.dispatch task 4585f453-94c4-477b-ad78-b3b0f12379d3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:25:55,276 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:25:55,352 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:25:55,403 DEBUG awx.main.dispatch task 4585f453-94c4-477b-ad78-b3b0f12379d3 is finished
2022-08-28 03:25:57,918 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a6d5d669-69da-4bc1-8220-b555c37866a2, queue=awx_private_queue)
[2022-08-28 03:25:57,918: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a6d5d669-69da-4bc1-8220-b555c37866a2, queue=awx_private_queue)
2022-08-28 03:25:57,935 DEBUG awx.main.dispatch delivered a6d5d669-69da-4bc1-8220-b555c37866a2 to worker[202] qsize 0
2022-08-28 03:25:57,938 DEBUG awx.main.dispatch task a6d5d669-69da-4bc1-8220-b555c37866a2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:25:57,955 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:25:57,960 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:25:27.957122+00:00
2022-08-28 03:25:58,008 DEBUG awx.main.dispatch task a6d5d669-69da-4bc1-8220-b555c37866a2 is finished
RESULT 2
OKREADY
2022-08-28 03:26:08,945 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(ade154ce-be9c-420d-9b39-f76cd81f58ca, queue=onap-awx-0)
[2022-08-28 03:26:08,945: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(ade154ce-be9c-420d-9b39-f76cd81f58ca, queue=onap-awx-0)
2022-08-28 03:26:09,222 DEBUG awx.main.dispatch delivered ade154ce-be9c-420d-9b39-f76cd81f58ca to worker[202] qsize 0
2022-08-28 03:26:09,226 DEBUG awx.main.dispatch task ade154ce-be9c-420d-9b39-f76cd81f58ca starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:26:09,227 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:26:09,295 DEBUG awx.main.dispatch task ade154ce-be9c-420d-9b39-f76cd81f58ca is finished
2022-08-28 03:26:15,210 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(4da9303a-15c8-4c3c-8fc8-3734ab92d2e6, queue=onap-awx-0)
[2022-08-28 03:26:15,210: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(4da9303a-15c8-4c3c-8fc8-3734ab92d2e6, queue=onap-awx-0)
2022-08-28 03:26:15,225 DEBUG awx.main.dispatch delivered 4da9303a-15c8-4c3c-8fc8-3734ab92d2e6 to worker[209] qsize 0
2022-08-28 03:26:15,228 DEBUG awx.main.dispatch task 4da9303a-15c8-4c3c-8fc8-3734ab92d2e6 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:26:15,250 DEBUG awx.main.dispatch task 4da9303a-15c8-4c3c-8fc8-3734ab92d2e6 is finished
2022-08-28 03:26:15,254 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(72e33adb-fbd7-4c24-b7e0-6d5f8c6680f8, queue=awx_private_queue)
[2022-08-28 03:26:15,254: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(72e33adb-fbd7-4c24-b7e0-6d5f8c6680f8, queue=awx_private_queue)
2022-08-28 03:26:15,271 DEBUG awx.main.dispatch delivered 72e33adb-fbd7-4c24-b7e0-6d5f8c6680f8 to worker[209] qsize 0
2022-08-28 03:26:15,273 DEBUG awx.main.dispatch task 72e33adb-fbd7-4c24-b7e0-6d5f8c6680f8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:26:15,275 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:26:15,298 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:26:15,337 DEBUG awx.main.dispatch task 72e33adb-fbd7-4c24-b7e0-6d5f8c6680f8 is finished
2022-08-28 03:26:27,921 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(50ee1e98-5b93-4c73-9be6-41417168cfe1, queue=awx_private_queue)
[2022-08-28 03:26:27,921: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(50ee1e98-5b93-4c73-9be6-41417168cfe1, queue=awx_private_queue)
2022-08-28 03:26:27,937 DEBUG awx.main.dispatch delivered 50ee1e98-5b93-4c73-9be6-41417168cfe1 to worker[207] qsize 0
2022-08-28 03:26:27,940 DEBUG awx.main.dispatch task 50ee1e98-5b93-4c73-9be6-41417168cfe1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:26:27,969 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:26:27,974 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:25:57.956129+00:00
2022-08-28 03:26:27,993 DEBUG awx.main.dispatch task 50ee1e98-5b93-4c73-9be6-41417168cfe1 is finished
2022-08-28 03:26:35,254 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0991d9df-37af-424c-8cbe-5c6879f05309, queue=awx_private_queue)
[2022-08-28 03:26:35,254: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0991d9df-37af-424c-8cbe-5c6879f05309, queue=awx_private_queue)
2022-08-28 03:26:35,271 DEBUG awx.main.dispatch delivered 0991d9df-37af-424c-8cbe-5c6879f05309 to worker[199] qsize 0
2022-08-28 03:26:35,273 DEBUG awx.main.dispatch task 0991d9df-37af-424c-8cbe-5c6879f05309 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:26:35,275 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:26:35,299 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:26:35,340 DEBUG awx.main.dispatch task 0991d9df-37af-424c-8cbe-5c6879f05309 is finished
2022-08-28 03:26:55,263 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f1a84eb8-8ac6-4126-ab04-646fd0f19443, queue=awx_private_queue)
[2022-08-28 03:26:55,263: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f1a84eb8-8ac6-4126-ab04-646fd0f19443, queue=awx_private_queue)
2022-08-28 03:26:55,279 DEBUG awx.main.dispatch delivered f1a84eb8-8ac6-4126-ab04-646fd0f19443 to worker[202] qsize 0
2022-08-28 03:26:55,281 DEBUG awx.main.dispatch task f1a84eb8-8ac6-4126-ab04-646fd0f19443 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:26:55,283 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:26:55,306 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:26:55,359 DEBUG awx.main.dispatch task f1a84eb8-8ac6-4126-ab04-646fd0f19443 is finished
2022-08-28 03:26:57,921 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d1a72420-d49a-4cac-ab12-ff96d790862c, queue=awx_private_queue)
[2022-08-28 03:26:57,921: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d1a72420-d49a-4cac-ab12-ff96d790862c, queue=awx_private_queue)
2022-08-28 03:26:57,950 DEBUG awx.main.dispatch delivered d1a72420-d49a-4cac-ab12-ff96d790862c to worker[199] qsize 0
2022-08-28 03:26:57,952 DEBUG awx.main.dispatch task d1a72420-d49a-4cac-ab12-ff96d790862c starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:26:57,974 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:26:57,978 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:26:27.969811+00:00
2022-08-28 03:26:58,006 DEBUG awx.main.dispatch task d1a72420-d49a-4cac-ab12-ff96d790862c is finished
RESULT 2
OKREADY
2022-08-28 03:27:08,945 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(496bd606-a9ee-43e4-851a-56a095b7ca60, queue=onap-awx-0)
[2022-08-28 03:27:08,945: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(496bd606-a9ee-43e4-851a-56a095b7ca60, queue=onap-awx-0)
2022-08-28 03:27:09,037 DEBUG awx.main.dispatch delivered 496bd606-a9ee-43e4-851a-56a095b7ca60 to worker[209] qsize 0
2022-08-28 03:27:09,039 DEBUG awx.main.dispatch task 496bd606-a9ee-43e4-851a-56a095b7ca60 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:27:09,040 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:27:09,133 DEBUG awx.main.dispatch task 496bd606-a9ee-43e4-851a-56a095b7ca60 is finished
2022-08-28 03:27:15,210 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(967fa645-1b0d-40d6-a29c-775079aa71d6, queue=onap-awx-0)
[2022-08-28 03:27:15,210: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(967fa645-1b0d-40d6-a29c-775079aa71d6, queue=onap-awx-0)
2022-08-28 03:27:15,424 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6cedca19-d235-4234-9a20-46d34848c5f6, queue=awx_private_queue)
[2022-08-28 03:27:15,424: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6cedca19-d235-4234-9a20-46d34848c5f6, queue=awx_private_queue)
2022-08-28 03:27:15,423 DEBUG awx.main.dispatch delivered 967fa645-1b0d-40d6-a29c-775079aa71d6 to worker[202] qsize 0
2022-08-28 03:27:15,426 DEBUG awx.main.dispatch task 967fa645-1b0d-40d6-a29c-775079aa71d6 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:27:15,440 DEBUG awx.main.dispatch delivered 6cedca19-d235-4234-9a20-46d34848c5f6 to worker[209] qsize 0
2022-08-28 03:27:15,443 DEBUG awx.main.dispatch task 6cedca19-d235-4234-9a20-46d34848c5f6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:27:15,445 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:27:15,447 DEBUG awx.main.dispatch task 967fa645-1b0d-40d6-a29c-775079aa71d6 is finished
2022-08-28 03:27:15,492 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:27:15,536 DEBUG awx.main.dispatch task 6cedca19-d235-4234-9a20-46d34848c5f6 is finished
2022-08-28 03:27:27,961 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e83497bf-ad39-49ec-84f0-0d7f22503224, queue=awx_private_queue)
[2022-08-28 03:27:27,961: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e83497bf-ad39-49ec-84f0-0d7f22503224, queue=awx_private_queue)
2022-08-28 03:27:27,977 DEBUG awx.main.dispatch delivered e83497bf-ad39-49ec-84f0-0d7f22503224 to worker[199] qsize 0
2022-08-28 03:27:27,980 DEBUG awx.main.dispatch task e83497bf-ad39-49ec-84f0-0d7f22503224 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:27:27,999 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:27:28,005 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:26:57.974352+00:00
2022-08-28 03:27:28,033 DEBUG awx.main.dispatch task e83497bf-ad39-49ec-84f0-0d7f22503224 is finished
2022-08-28 03:27:35,424 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ca2302a0-b060-47ff-86df-d9cbfb66a2c6, queue=awx_private_queue)
[2022-08-28 03:27:35,424: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ca2302a0-b060-47ff-86df-d9cbfb66a2c6, queue=awx_private_queue)
2022-08-28 03:27:35,445 DEBUG awx.main.dispatch delivered ca2302a0-b060-47ff-86df-d9cbfb66a2c6 to worker[209] qsize 0
2022-08-28 03:27:35,447 DEBUG awx.main.dispatch task ca2302a0-b060-47ff-86df-d9cbfb66a2c6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:27:35,449 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:27:35,472 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:27:35,508 DEBUG awx.main.dispatch task ca2302a0-b060-47ff-86df-d9cbfb66a2c6 is finished
2022-08-28 03:27:55,423 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(08021d6e-eebf-42e2-b301-13a4c5164f3c, queue=awx_private_queue)
[2022-08-28 03:27:55,423: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(08021d6e-eebf-42e2-b301-13a4c5164f3c, queue=awx_private_queue)
2022-08-28 03:27:55,438 DEBUG awx.main.dispatch delivered 08021d6e-eebf-42e2-b301-13a4c5164f3c to worker[202] qsize 0
2022-08-28 03:27:55,441 DEBUG awx.main.dispatch task 08021d6e-eebf-42e2-b301-13a4c5164f3c starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:27:55,444 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:27:55,475 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:27:55,523 DEBUG awx.main.dispatch task 08021d6e-eebf-42e2-b301-13a4c5164f3c is finished
2022-08-28 03:27:57,960 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(fd303bac-ae19-4f4f-8978-156e9f67cb3e, queue=awx_private_queue)
[2022-08-28 03:27:57,960: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(fd303bac-ae19-4f4f-8978-156e9f67cb3e, queue=awx_private_queue)
2022-08-28 03:27:57,977 DEBUG awx.main.dispatch delivered fd303bac-ae19-4f4f-8978-156e9f67cb3e to worker[202] qsize 0
2022-08-28 03:27:57,979 DEBUG awx.main.dispatch task fd303bac-ae19-4f4f-8978-156e9f67cb3e starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:27:58,000 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:27:58,003 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:27:27.999939+00:00
2022-08-28 03:27:58,051 DEBUG awx.main.dispatch task fd303bac-ae19-4f4f-8978-156e9f67cb3e is finished
RESULT 2
OKREADY
2022-08-28 03:28:08,948 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8f2b8a49-0b9b-4852-b3e3-decaad8f594c, queue=onap-awx-0)
[2022-08-28 03:28:08,948: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8f2b8a49-0b9b-4852-b3e3-decaad8f594c, queue=onap-awx-0)
2022-08-28 03:28:09,014 DEBUG awx.main.dispatch delivered 8f2b8a49-0b9b-4852-b3e3-decaad8f594c to worker[199] qsize 0
2022-08-28 03:28:09,018 DEBUG awx.main.dispatch task 8f2b8a49-0b9b-4852-b3e3-decaad8f594c starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:28:09,021 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:28:09,122 DEBUG awx.main.dispatch task 8f2b8a49-0b9b-4852-b3e3-decaad8f594c is finished
2022-08-28 03:28:15,210 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(914ceac5-4f75-4ef3-b788-f93e8788acbc, queue=onap-awx-0)
[2022-08-28 03:28:15,210: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(914ceac5-4f75-4ef3-b788-f93e8788acbc, queue=onap-awx-0)
2022-08-28 03:28:15,448 DEBUG awx.main.dispatch delivered 914ceac5-4f75-4ef3-b788-f93e8788acbc to worker[207] qsize 0
2022-08-28 03:28:15,449 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3cd43787-f408-4baf-a268-864d3d7e0ba3, queue=awx_private_queue)
[2022-08-28 03:28:15,449: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3cd43787-f408-4baf-a268-864d3d7e0ba3, queue=awx_private_queue)
2022-08-28 03:28:15,451 DEBUG awx.main.dispatch task 914ceac5-4f75-4ef3-b788-f93e8788acbc starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:28:15,464 DEBUG awx.main.dispatch delivered 3cd43787-f408-4baf-a268-864d3d7e0ba3 to worker[199] qsize 0
2022-08-28 03:28:15,466 DEBUG awx.main.dispatch task 3cd43787-f408-4baf-a268-864d3d7e0ba3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:28:15,467 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:28:15,482 DEBUG awx.main.dispatch task 914ceac5-4f75-4ef3-b788-f93e8788acbc is finished
2022-08-28 03:28:15,512 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:28:15,575 DEBUG awx.main.dispatch task 3cd43787-f408-4baf-a268-864d3d7e0ba3 is finished
2022-08-28 03:28:27,963 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e879fb0f-2dd6-4738-a226-707bcddbfe92, queue=awx_private_queue)
[2022-08-28 03:28:27,963: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e879fb0f-2dd6-4738-a226-707bcddbfe92, queue=awx_private_queue)
2022-08-28 03:28:27,978 DEBUG awx.main.dispatch delivered e879fb0f-2dd6-4738-a226-707bcddbfe92 to worker[209] qsize 0
2022-08-28 03:28:27,981 DEBUG awx.main.dispatch task e879fb0f-2dd6-4738-a226-707bcddbfe92 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:28:27,997 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:28:28,001 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:27:58.000557+00:00
2022-08-28 03:28:28,019 DEBUG awx.main.dispatch task e879fb0f-2dd6-4738-a226-707bcddbfe92 is finished
2022-08-28 03:28:35,450 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(04aa6e79-a4f6-426d-8aea-f579480b9a08, queue=awx_private_queue)
[2022-08-28 03:28:35,450: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(04aa6e79-a4f6-426d-8aea-f579480b9a08, queue=awx_private_queue)
2022-08-28 03:28:35,468 DEBUG awx.main.dispatch delivered 04aa6e79-a4f6-426d-8aea-f579480b9a08 to worker[207] qsize 0
2022-08-28 03:28:35,471 DEBUG awx.main.dispatch task 04aa6e79-a4f6-426d-8aea-f579480b9a08 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:28:35,472 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:28:35,500 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:28:35,546 DEBUG awx.main.dispatch task 04aa6e79-a4f6-426d-8aea-f579480b9a08 is finished
2022-08-28 03:28:55,449 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(39196117-30ca-4dad-98e8-f6f5bf594292, queue=awx_private_queue)
[2022-08-28 03:28:55,449: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(39196117-30ca-4dad-98e8-f6f5bf594292, queue=awx_private_queue)
2022-08-28 03:28:55,466 DEBUG awx.main.dispatch delivered 39196117-30ca-4dad-98e8-f6f5bf594292 to worker[202] qsize 0
2022-08-28 03:28:55,470 DEBUG awx.main.dispatch task 39196117-30ca-4dad-98e8-f6f5bf594292 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:28:55,474 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:28:55,501 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:28:55,539 DEBUG awx.main.dispatch task 39196117-30ca-4dad-98e8-f6f5bf594292 is finished
2022-08-28 03:28:57,963 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c93baf6e-5d60-4ec8-806b-9018d99f5c1d, queue=awx_private_queue)
[2022-08-28 03:28:57,963: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c93baf6e-5d60-4ec8-806b-9018d99f5c1d, queue=awx_private_queue)
2022-08-28 03:28:57,980 DEBUG awx.main.dispatch delivered c93baf6e-5d60-4ec8-806b-9018d99f5c1d to worker[209] qsize 0
2022-08-28 03:28:57,983 DEBUG awx.main.dispatch task c93baf6e-5d60-4ec8-806b-9018d99f5c1d starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:28:58,010 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:28:58,014 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:28:27.997506+00:00
2022-08-28 03:28:58,064 DEBUG awx.main.dispatch task c93baf6e-5d60-4ec8-806b-9018d99f5c1d is finished
RESULT 2
OKREADY
2022-08-28 03:29:08,947 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(425f0caf-6979-45dd-b00c-caa874a78382, queue=onap-awx-0)
[2022-08-28 03:29:08,947: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(425f0caf-6979-45dd-b00c-caa874a78382, queue=onap-awx-0)
2022-08-28 03:29:08,988 DEBUG awx.main.dispatch delivered 425f0caf-6979-45dd-b00c-caa874a78382 to worker[199] qsize 0
2022-08-28 03:29:08,991 DEBUG awx.main.dispatch task 425f0caf-6979-45dd-b00c-caa874a78382 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:29:08,992 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:29:09,119 DEBUG awx.main.dispatch task 425f0caf-6979-45dd-b00c-caa874a78382 is finished
2022-08-28 03:29:15,210 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d1c02dbd-4a12-4627-a982-d334d27fdc2f, queue=onap-awx-0)
[2022-08-28 03:29:15,210: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d1c02dbd-4a12-4627-a982-d334d27fdc2f, queue=onap-awx-0)
2022-08-28 03:29:15,257 DEBUG awx.main.dispatch delivered d1c02dbd-4a12-4627-a982-d334d27fdc2f to worker[209] qsize 0
2022-08-28 03:29:15,260 DEBUG awx.main.dispatch task d1c02dbd-4a12-4627-a982-d334d27fdc2f starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:29:15,275 DEBUG awx.main.dispatch task d1c02dbd-4a12-4627-a982-d334d27fdc2f is finished
2022-08-28 03:29:15,450 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cc52db91-e3da-4d39-a03d-81af31ff0a03, queue=awx_private_queue)
[2022-08-28 03:29:15,450: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cc52db91-e3da-4d39-a03d-81af31ff0a03, queue=awx_private_queue)
2022-08-28 03:29:15,683 DEBUG awx.main.dispatch delivered cc52db91-e3da-4d39-a03d-81af31ff0a03 to worker[207] qsize 0
2022-08-28 03:29:15,686 DEBUG awx.main.dispatch task cc52db91-e3da-4d39-a03d-81af31ff0a03 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:29:15,689 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:29:15,714 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:29:15,829 DEBUG awx.main.dispatch task cc52db91-e3da-4d39-a03d-81af31ff0a03 is finished
2022-08-28 03:29:27,967 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(59c2108c-da39-4d08-aeb6-61831e4747bd, queue=awx_private_queue)
[2022-08-28 03:29:27,967: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(59c2108c-da39-4d08-aeb6-61831e4747bd, queue=awx_private_queue)
2022-08-28 03:29:27,987 DEBUG awx.main.dispatch delivered 59c2108c-da39-4d08-aeb6-61831e4747bd to worker[199] qsize 0
2022-08-28 03:29:27,989 DEBUG awx.main.dispatch task 59c2108c-da39-4d08-aeb6-61831e4747bd starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:29:28,007 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:29:28,011 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:28:58.010660+00:00
2022-08-28 03:29:28,038 DEBUG awx.main.dispatch task 59c2108c-da39-4d08-aeb6-61831e4747bd is finished
2022-08-28 03:29:35,449 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b3e083ce-f535-4ffa-94a3-c996b4ffd1d7, queue=awx_private_queue)
[2022-08-28 03:29:35,449: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b3e083ce-f535-4ffa-94a3-c996b4ffd1d7, queue=awx_private_queue)
2022-08-28 03:29:35,465 DEBUG awx.main.dispatch delivered b3e083ce-f535-4ffa-94a3-c996b4ffd1d7 to worker[209] qsize 0
2022-08-28 03:29:35,468 DEBUG awx.main.dispatch task b3e083ce-f535-4ffa-94a3-c996b4ffd1d7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:29:35,471 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:29:35,509 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:29:35,556 DEBUG awx.main.dispatch task b3e083ce-f535-4ffa-94a3-c996b4ffd1d7 is finished
2022-08-28 03:29:55,460 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a21c5fa0-9b5a-4182-91f2-d3b2f1679b6b, queue=awx_private_queue)
[2022-08-28 03:29:55,460: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a21c5fa0-9b5a-4182-91f2-d3b2f1679b6b, queue=awx_private_queue)
2022-08-28 03:29:55,475 DEBUG awx.main.dispatch delivered a21c5fa0-9b5a-4182-91f2-d3b2f1679b6b to worker[199] qsize 0
2022-08-28 03:29:55,477 DEBUG awx.main.dispatch task a21c5fa0-9b5a-4182-91f2-d3b2f1679b6b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:29:55,478 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:29:55,533 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:29:55,597 DEBUG awx.main.dispatch task a21c5fa0-9b5a-4182-91f2-d3b2f1679b6b is finished
2022-08-28 03:29:57,966 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(efef5f9b-6ab5-463a-b4d7-57e7b9233efd, queue=awx_private_queue)
[2022-08-28 03:29:57,966: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(efef5f9b-6ab5-463a-b4d7-57e7b9233efd, queue=awx_private_queue)
2022-08-28 03:29:57,982 DEBUG awx.main.dispatch delivered efef5f9b-6ab5-463a-b4d7-57e7b9233efd to worker[209] qsize 0
2022-08-28 03:29:57,984 DEBUG awx.main.dispatch task efef5f9b-6ab5-463a-b4d7-57e7b9233efd starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:29:58,002 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:29:58,006 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:29:28.007890+00:00
2022-08-28 03:29:58,041 DEBUG awx.main.dispatch task efef5f9b-6ab5-463a-b4d7-57e7b9233efd is finished
RESULT 2
OKREADY
2022-08-28 03:30:08,949 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(3cbfaa67-8f14-46b5-922d-c155bd8e7188, queue=onap-awx-0)
[2022-08-28 03:30:08,949: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(3cbfaa67-8f14-46b5-922d-c155bd8e7188, queue=onap-awx-0)
2022-08-28 03:30:09,004 DEBUG awx.main.dispatch delivered 3cbfaa67-8f14-46b5-922d-c155bd8e7188 to worker[202] qsize 0
2022-08-28 03:30:09,007 DEBUG awx.main.dispatch task 3cbfaa67-8f14-46b5-922d-c155bd8e7188 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:30:09,009 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:30:09,210 DEBUG awx.main.dispatch task 3cbfaa67-8f14-46b5-922d-c155bd8e7188 is finished
2022-08-28 03:30:15,210 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f22917ae-f2b4-46bd-a3fa-02c79a548bd2, queue=onap-awx-0)
[2022-08-28 03:30:15,210: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f22917ae-f2b4-46bd-a3fa-02c79a548bd2, queue=onap-awx-0)
2022-08-28 03:30:15,241 DEBUG awx.main.dispatch delivered f22917ae-f2b4-46bd-a3fa-02c79a548bd2 to worker[207] qsize 0
2022-08-28 03:30:15,243 DEBUG awx.main.dispatch task f22917ae-f2b4-46bd-a3fa-02c79a548bd2 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:30:15,259 DEBUG awx.main.dispatch task f22917ae-f2b4-46bd-a3fa-02c79a548bd2 is finished
2022-08-28 03:30:15,460 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(aeb46f95-3ffb-425e-80e7-ee76de575046, queue=awx_private_queue)
[2022-08-28 03:30:15,460: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(aeb46f95-3ffb-425e-80e7-ee76de575046, queue=awx_private_queue)
2022-08-28 03:30:15,693 DEBUG awx.main.dispatch delivered aeb46f95-3ffb-425e-80e7-ee76de575046 to worker[209] qsize 0
2022-08-28 03:30:15,696 DEBUG awx.main.dispatch task aeb46f95-3ffb-425e-80e7-ee76de575046 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:30:15,697 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:30:15,790 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:30:15,945 DEBUG awx.main.dispatch task aeb46f95-3ffb-425e-80e7-ee76de575046 is finished
2022-08-28 03:30:27,967 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e667413d-7c9b-4de8-a2d3-447cf55d7ae1, queue=awx_private_queue)
[2022-08-28 03:30:27,967: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e667413d-7c9b-4de8-a2d3-447cf55d7ae1, queue=awx_private_queue)
2022-08-28 03:30:27,982 DEBUG awx.main.dispatch delivered e667413d-7c9b-4de8-a2d3-447cf55d7ae1 to worker[207] qsize 0
2022-08-28 03:30:27,984 DEBUG awx.main.dispatch task e667413d-7c9b-4de8-a2d3-447cf55d7ae1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:30:28,078 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:30:28,088 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:29:58.003125+00:00
2022-08-28 03:30:28,154 DEBUG awx.main.dispatch task e667413d-7c9b-4de8-a2d3-447cf55d7ae1 is finished
2022-08-28 03:30:35,460 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1774a69f-c6b4-4cab-a40d-d18aa57f0e6d, queue=awx_private_queue)
[2022-08-28 03:30:35,460: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1774a69f-c6b4-4cab-a40d-d18aa57f0e6d, queue=awx_private_queue)
2022-08-28 03:30:35,476 DEBUG awx.main.dispatch delivered 1774a69f-c6b4-4cab-a40d-d18aa57f0e6d to worker[209] qsize 0
2022-08-28 03:30:35,478 DEBUG awx.main.dispatch task 1774a69f-c6b4-4cab-a40d-d18aa57f0e6d starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:30:35,481 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:30:35,520 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:30:35,562 DEBUG awx.main.dispatch task 1774a69f-c6b4-4cab-a40d-d18aa57f0e6d is finished
2022-08-28 03:30:55,473 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2cd289c7-961e-4df3-bb1f-b6d53df5a245, queue=awx_private_queue)
[2022-08-28 03:30:55,473: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2cd289c7-961e-4df3-bb1f-b6d53df5a245, queue=awx_private_queue)
2022-08-28 03:30:55,491 DEBUG awx.main.dispatch delivered 2cd289c7-961e-4df3-bb1f-b6d53df5a245 to worker[202] qsize 0
2022-08-28 03:30:55,494 DEBUG awx.main.dispatch task 2cd289c7-961e-4df3-bb1f-b6d53df5a245 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:30:55,496 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:30:55,523 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:30:55,564 DEBUG awx.main.dispatch task 2cd289c7-961e-4df3-bb1f-b6d53df5a245 is finished
2022-08-28 03:30:57,966 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1f7e80b6-15e9-49c7-a25c-a2685903c96a, queue=awx_private_queue)
[2022-08-28 03:30:57,966: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1f7e80b6-15e9-49c7-a25c-a2685903c96a, queue=awx_private_queue)
2022-08-28 03:30:57,983 DEBUG awx.main.dispatch delivered 1f7e80b6-15e9-49c7-a25c-a2685903c96a to worker[209] qsize 0
2022-08-28 03:30:57,985 DEBUG awx.main.dispatch task 1f7e80b6-15e9-49c7-a25c-a2685903c96a starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:30:58,022 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:30:58,026 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:30:28.078959+00:00
2022-08-28 03:30:58,062 DEBUG awx.main.dispatch task 1f7e80b6-15e9-49c7-a25c-a2685903c96a is finished
RESULT 2
OKREADY
2022-08-28 03:31:08,952 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e1fcba3c-5d93-4730-a9d9-0c4e5f229481, queue=onap-awx-0)
[2022-08-28 03:31:08,952: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e1fcba3c-5d93-4730-a9d9-0c4e5f229481, queue=onap-awx-0)
2022-08-28 03:31:08,993 DEBUG awx.main.dispatch delivered e1fcba3c-5d93-4730-a9d9-0c4e5f229481 to worker[202] qsize 0
2022-08-28 03:31:08,996 DEBUG awx.main.dispatch task e1fcba3c-5d93-4730-a9d9-0c4e5f229481 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:31:08,997 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:31:09,028 DEBUG awx.main.dispatch task e1fcba3c-5d93-4730-a9d9-0c4e5f229481 is finished
2022-08-28 03:31:15,211 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(77c92611-6946-4ea3-aba9-fc1b55e63fdc, queue=onap-awx-0)
[2022-08-28 03:31:15,211: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(77c92611-6946-4ea3-aba9-fc1b55e63fdc, queue=onap-awx-0)
2022-08-28 03:31:15,243 DEBUG awx.main.dispatch delivered 77c92611-6946-4ea3-aba9-fc1b55e63fdc to worker[202] qsize 0
2022-08-28 03:31:15,246 DEBUG awx.main.dispatch task 77c92611-6946-4ea3-aba9-fc1b55e63fdc starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:31:15,259 DEBUG awx.main.dispatch task 77c92611-6946-4ea3-aba9-fc1b55e63fdc is finished
2022-08-28 03:31:15,472 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7beec044-991e-43ef-9983-f1e4432b6ff4, queue=awx_private_queue)
[2022-08-28 03:31:15,472: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7beec044-991e-43ef-9983-f1e4432b6ff4, queue=awx_private_queue)
2022-08-28 03:31:15,712 DEBUG awx.main.dispatch delivered 7beec044-991e-43ef-9983-f1e4432b6ff4 to worker[209] qsize 0
2022-08-28 03:31:15,716 DEBUG awx.main.dispatch task 7beec044-991e-43ef-9983-f1e4432b6ff4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:31:15,717 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:31:15,741 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:31:15,792 DEBUG awx.main.dispatch task 7beec044-991e-43ef-9983-f1e4432b6ff4 is finished
2022-08-28 03:31:27,967 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a09158cc-0947-4d9a-a5c5-aea7f5c4f9b7, queue=awx_private_queue)
[2022-08-28 03:31:27,967: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a09158cc-0947-4d9a-a5c5-aea7f5c4f9b7, queue=awx_private_queue)
2022-08-28 03:31:27,983 DEBUG awx.main.dispatch delivered a09158cc-0947-4d9a-a5c5-aea7f5c4f9b7 to worker[199] qsize 0
2022-08-28 03:31:27,986 DEBUG awx.main.dispatch task a09158cc-0947-4d9a-a5c5-aea7f5c4f9b7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:31:28,034 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:31:28,045 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:30:58.022275+00:00
2022-08-28 03:31:28,107 DEBUG awx.main.dispatch task a09158cc-0947-4d9a-a5c5-aea7f5c4f9b7 is finished
2022-08-28 03:31:35,472 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bc201131-006b-4a6a-a986-684159ea25f3, queue=awx_private_queue)
[2022-08-28 03:31:35,472: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bc201131-006b-4a6a-a986-684159ea25f3, queue=awx_private_queue)
2022-08-28 03:31:35,489 DEBUG awx.main.dispatch delivered bc201131-006b-4a6a-a986-684159ea25f3 to worker[209] qsize 0
2022-08-28 03:31:35,492 DEBUG awx.main.dispatch task bc201131-006b-4a6a-a986-684159ea25f3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:31:35,494 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:31:35,524 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:31:35,589 DEBUG awx.main.dispatch task bc201131-006b-4a6a-a986-684159ea25f3 is finished
2022-08-28 03:31:55,475 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fc5fd49d-8811-4631-b084-5c4025b9a7a8, queue=awx_private_queue)
[2022-08-28 03:31:55,475: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fc5fd49d-8811-4631-b084-5c4025b9a7a8, queue=awx_private_queue)
2022-08-28 03:31:55,494 DEBUG awx.main.dispatch delivered fc5fd49d-8811-4631-b084-5c4025b9a7a8 to worker[202] qsize 0
2022-08-28 03:31:55,497 DEBUG awx.main.dispatch task fc5fd49d-8811-4631-b084-5c4025b9a7a8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:31:55,498 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:31:55,521 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:31:55,588 DEBUG awx.main.dispatch task fc5fd49d-8811-4631-b084-5c4025b9a7a8 is finished
2022-08-28 03:31:56,257 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(bbe31aab-f92a-452a-883a-2e33cf88f201, queue=onap-awx-0)
[2022-08-28 03:31:56,257: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(bbe31aab-f92a-452a-883a-2e33cf88f201, queue=onap-awx-0)
2022-08-28 03:31:56,273 DEBUG awx.main.dispatch delivered bbe31aab-f92a-452a-883a-2e33cf88f201 to worker[202] qsize 0
2022-08-28 03:31:56,276 DEBUG awx.main.dispatch task bbe31aab-f92a-452a-883a-2e33cf88f201 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2022-08-28 03:31:56,277 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2022-08-28 03:31:56,308 DEBUG awx.main.dispatch task bbe31aab-f92a-452a-883a-2e33cf88f201 is finished
2022-08-28 03:31:57,967 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(211200f2-82b2-41ca-b0f0-8cf99de4abb0, queue=awx_private_queue)
[2022-08-28 03:31:57,967: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(211200f2-82b2-41ca-b0f0-8cf99de4abb0, queue=awx_private_queue)
2022-08-28 03:31:57,982 DEBUG awx.main.dispatch delivered 211200f2-82b2-41ca-b0f0-8cf99de4abb0 to worker[202] qsize 0
2022-08-28 03:31:57,985 DEBUG awx.main.dispatch task 211200f2-82b2-41ca-b0f0-8cf99de4abb0 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:31:58,032 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:31:58,042 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:31:28.034743+00:00
2022-08-28 03:31:58,071 DEBUG awx.main.dispatch task 211200f2-82b2-41ca-b0f0-8cf99de4abb0 is finished
RESULT 2
OKREADY
2022-08-28 03:32:08,952 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(2dc9c759-610e-4a78-a237-8ed8e0238c84, queue=onap-awx-0)
[2022-08-28 03:32:08,952: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(2dc9c759-610e-4a78-a237-8ed8e0238c84, queue=onap-awx-0)
2022-08-28 03:32:08,990 DEBUG awx.main.dispatch delivered 2dc9c759-610e-4a78-a237-8ed8e0238c84 to worker[207] qsize 0
2022-08-28 03:32:08,993 DEBUG awx.main.dispatch task 2dc9c759-610e-4a78-a237-8ed8e0238c84 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:32:08,996 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:32:09,028 DEBUG awx.main.dispatch task 2dc9c759-610e-4a78-a237-8ed8e0238c84 is finished
2022-08-28 03:32:15,210 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3cebf3b8-341a-4ef8-ac4e-0306b4c15428, queue=onap-awx-0)
[2022-08-28 03:32:15,210: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3cebf3b8-341a-4ef8-ac4e-0306b4c15428, queue=onap-awx-0)
2022-08-28 03:32:15,263 DEBUG awx.main.dispatch delivered 3cebf3b8-341a-4ef8-ac4e-0306b4c15428 to worker[199] qsize 0
2022-08-28 03:32:15,265 DEBUG awx.main.dispatch task 3cebf3b8-341a-4ef8-ac4e-0306b4c15428 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:32:15,284 DEBUG awx.main.dispatch task 3cebf3b8-341a-4ef8-ac4e-0306b4c15428 is finished
2022-08-28 03:32:15,475 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a2dcbe56-5be2-4072-b814-a4acfb9744bb, queue=awx_private_queue)
[2022-08-28 03:32:15,475: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a2dcbe56-5be2-4072-b814-a4acfb9744bb, queue=awx_private_queue)
2022-08-28 03:32:15,692 DEBUG awx.main.dispatch delivered a2dcbe56-5be2-4072-b814-a4acfb9744bb to worker[209] qsize 0
2022-08-28 03:32:15,694 DEBUG awx.main.dispatch task a2dcbe56-5be2-4072-b814-a4acfb9744bb starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:32:15,696 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:32:15,724 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:32:15,769 DEBUG awx.main.dispatch task a2dcbe56-5be2-4072-b814-a4acfb9744bb is finished
2022-08-28 03:32:27,970 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c7e70497-9396-4ca3-b01f-e1664651337e, queue=awx_private_queue)
[2022-08-28 03:32:27,970: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c7e70497-9396-4ca3-b01f-e1664651337e, queue=awx_private_queue)
2022-08-28 03:32:27,986 DEBUG awx.main.dispatch delivered c7e70497-9396-4ca3-b01f-e1664651337e to worker[207] qsize 0
2022-08-28 03:32:27,989 DEBUG awx.main.dispatch task c7e70497-9396-4ca3-b01f-e1664651337e starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:32:28,024 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:32:28,030 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:31:58.032586+00:00
2022-08-28 03:32:28,055 DEBUG awx.main.dispatch task c7e70497-9396-4ca3-b01f-e1664651337e is finished
2022-08-28 03:32:35,475 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c57b53a5-3f67-4155-aa34-9be793b235f7, queue=awx_private_queue)
[2022-08-28 03:32:35,475: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c57b53a5-3f67-4155-aa34-9be793b235f7, queue=awx_private_queue)
2022-08-28 03:32:35,490 DEBUG awx.main.dispatch delivered c57b53a5-3f67-4155-aa34-9be793b235f7 to worker[202] qsize 0
2022-08-28 03:32:35,493 DEBUG awx.main.dispatch task c57b53a5-3f67-4155-aa34-9be793b235f7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:32:35,495 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:32:35,521 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:32:35,560 DEBUG awx.main.dispatch task c57b53a5-3f67-4155-aa34-9be793b235f7 is finished
2022-08-28 03:32:55,475 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(902b0add-96ec-4237-afdc-96d7215b6b25, queue=awx_private_queue)
[2022-08-28 03:32:55,475: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(902b0add-96ec-4237-afdc-96d7215b6b25, queue=awx_private_queue)
2022-08-28 03:32:55,491 DEBUG awx.main.dispatch delivered 902b0add-96ec-4237-afdc-96d7215b6b25 to worker[199] qsize 0
2022-08-28 03:32:55,494 DEBUG awx.main.dispatch task 902b0add-96ec-4237-afdc-96d7215b6b25 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:32:55,497 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:32:55,520 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:32:55,560 DEBUG awx.main.dispatch task 902b0add-96ec-4237-afdc-96d7215b6b25 is finished
2022-08-28 03:32:57,969 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8b75f57a-e3e8-4553-9efe-b4708feac93d, queue=awx_private_queue)
[2022-08-28 03:32:57,969: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8b75f57a-e3e8-4553-9efe-b4708feac93d, queue=awx_private_queue)
2022-08-28 03:32:57,984 DEBUG awx.main.dispatch delivered 8b75f57a-e3e8-4553-9efe-b4708feac93d to worker[207] qsize 0
2022-08-28 03:32:57,987 DEBUG awx.main.dispatch task 8b75f57a-e3e8-4553-9efe-b4708feac93d starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:32:58,023 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:32:58,033 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:32:28.024646+00:00
2022-08-28 03:32:58,059 DEBUG awx.main.dispatch task 8b75f57a-e3e8-4553-9efe-b4708feac93d is finished
RESULT 2
OKREADY
2022-08-28 03:33:08,954 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d17907c9-2d2f-432f-863e-f8623ff90a41, queue=onap-awx-0)
[2022-08-28 03:33:08,954: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d17907c9-2d2f-432f-863e-f8623ff90a41, queue=onap-awx-0)
2022-08-28 03:33:09,003 DEBUG awx.main.dispatch delivered d17907c9-2d2f-432f-863e-f8623ff90a41 to worker[199] qsize 0
2022-08-28 03:33:09,005 DEBUG awx.main.dispatch task d17907c9-2d2f-432f-863e-f8623ff90a41 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:33:09,007 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:33:09,040 DEBUG awx.main.dispatch task d17907c9-2d2f-432f-863e-f8623ff90a41 is finished
2022-08-28 03:33:15,210 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(95e165c4-fa33-41f5-bbb0-1cd5aa324b21, queue=onap-awx-0)
[2022-08-28 03:33:15,210: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(95e165c4-fa33-41f5-bbb0-1cd5aa324b21, queue=onap-awx-0)
2022-08-28 03:33:15,244 DEBUG awx.main.dispatch delivered 95e165c4-fa33-41f5-bbb0-1cd5aa324b21 to worker[209] qsize 0
2022-08-28 03:33:15,247 DEBUG awx.main.dispatch task 95e165c4-fa33-41f5-bbb0-1cd5aa324b21 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:33:15,475 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2253ed4e-1e82-4758-94cc-7d0f201ad0cf, queue=awx_private_queue)
[2022-08-28 03:33:15,475: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2253ed4e-1e82-4758-94cc-7d0f201ad0cf, queue=awx_private_queue)
2022-08-28 03:33:15,482 DEBUG awx.main.dispatch task 95e165c4-fa33-41f5-bbb0-1cd5aa324b21 is finished
2022-08-28 03:33:15,492 DEBUG awx.main.dispatch delivered 2253ed4e-1e82-4758-94cc-7d0f201ad0cf to worker[202] qsize 0
2022-08-28 03:33:15,494 DEBUG awx.main.dispatch task 2253ed4e-1e82-4758-94cc-7d0f201ad0cf starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:33:15,496 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:33:15,530 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:33:15,630 DEBUG awx.main.dispatch task 2253ed4e-1e82-4758-94cc-7d0f201ad0cf is finished
2022-08-28 03:33:27,977 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(cc47e26d-22b0-46f4-b322-0f3200e5b1ed, queue=awx_private_queue)
[2022-08-28 03:33:27,977: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(cc47e26d-22b0-46f4-b322-0f3200e5b1ed, queue=awx_private_queue)
2022-08-28 03:33:27,996 DEBUG awx.main.dispatch delivered cc47e26d-22b0-46f4-b322-0f3200e5b1ed to worker[199] qsize 0
2022-08-28 03:33:27,999 DEBUG awx.main.dispatch task cc47e26d-22b0-46f4-b322-0f3200e5b1ed starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:33:28,026 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:33:28,030 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:32:58.023916+00:00
2022-08-28 03:33:28,077 DEBUG awx.main.dispatch task cc47e26d-22b0-46f4-b322-0f3200e5b1ed is finished
2022-08-28 03:33:35,475 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f2f8c767-62fb-462c-81b1-8d3e019ef2d9, queue=awx_private_queue)
[2022-08-28 03:33:35,475: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f2f8c767-62fb-462c-81b1-8d3e019ef2d9, queue=awx_private_queue)
2022-08-28 03:33:35,492 DEBUG awx.main.dispatch delivered f2f8c767-62fb-462c-81b1-8d3e019ef2d9 to worker[207] qsize 0
2022-08-28 03:33:35,495 DEBUG awx.main.dispatch task f2f8c767-62fb-462c-81b1-8d3e019ef2d9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:33:35,498 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:33:35,524 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:33:35,565 DEBUG awx.main.dispatch task f2f8c767-62fb-462c-81b1-8d3e019ef2d9 is finished
2022-08-28 03:33:55,475 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(63c1dc14-c5ae-4d80-bcf2-525978641e91, queue=awx_private_queue)
[2022-08-28 03:33:55,475: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(63c1dc14-c5ae-4d80-bcf2-525978641e91, queue=awx_private_queue)
2022-08-28 03:33:55,491 DEBUG awx.main.dispatch delivered 63c1dc14-c5ae-4d80-bcf2-525978641e91 to worker[209] qsize 0
2022-08-28 03:33:55,493 DEBUG awx.main.dispatch task 63c1dc14-c5ae-4d80-bcf2-525978641e91 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:33:55,494 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:33:55,528 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:33:55,571 DEBUG awx.main.dispatch task 63c1dc14-c5ae-4d80-bcf2-525978641e91 is finished
2022-08-28 03:33:57,977 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2e99d878-13a5-41c7-aedf-dc8b09315bfa, queue=awx_private_queue)
[2022-08-28 03:33:57,977: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2e99d878-13a5-41c7-aedf-dc8b09315bfa, queue=awx_private_queue)
2022-08-28 03:33:57,994 DEBUG awx.main.dispatch delivered 2e99d878-13a5-41c7-aedf-dc8b09315bfa to worker[207] qsize 0
2022-08-28 03:33:57,996 DEBUG awx.main.dispatch task 2e99d878-13a5-41c7-aedf-dc8b09315bfa starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:33:58,030 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:33:58,045 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:33:28.026892+00:00
2022-08-28 03:33:58,121 DEBUG awx.main.dispatch task 2e99d878-13a5-41c7-aedf-dc8b09315bfa is finished
RESULT 2
OKREADY
2022-08-28 03:34:08,955 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e37fea81-32b5-473e-979d-271c9e6d81df, queue=onap-awx-0)
[2022-08-28 03:34:08,955: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e37fea81-32b5-473e-979d-271c9e6d81df, queue=onap-awx-0)
2022-08-28 03:34:08,989 DEBUG awx.main.dispatch delivered e37fea81-32b5-473e-979d-271c9e6d81df to worker[202] qsize 0
2022-08-28 03:34:08,992 DEBUG awx.main.dispatch task e37fea81-32b5-473e-979d-271c9e6d81df starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:34:08,993 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:34:09,035 DEBUG awx.main.dispatch task e37fea81-32b5-473e-979d-271c9e6d81df is finished
2022-08-28 03:34:15,211 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a2153793-c45f-4d16-8ec2-cf24b470b565, queue=onap-awx-0)
[2022-08-28 03:34:15,211: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a2153793-c45f-4d16-8ec2-cf24b470b565, queue=onap-awx-0)
2022-08-28 03:34:15,246 DEBUG awx.main.dispatch delivered a2153793-c45f-4d16-8ec2-cf24b470b565 to worker[202] qsize 0
2022-08-28 03:34:15,248 DEBUG awx.main.dispatch task a2153793-c45f-4d16-8ec2-cf24b470b565 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:34:15,269 DEBUG awx.main.dispatch task a2153793-c45f-4d16-8ec2-cf24b470b565 is finished
2022-08-28 03:34:15,475 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1d758e98-70ae-4db0-aa7b-a43fa4ff901d, queue=awx_private_queue)
[2022-08-28 03:34:15,475: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1d758e98-70ae-4db0-aa7b-a43fa4ff901d, queue=awx_private_queue)
2022-08-28 03:34:15,705 DEBUG awx.main.dispatch delivered 1d758e98-70ae-4db0-aa7b-a43fa4ff901d to worker[209] qsize 0
2022-08-28 03:34:15,707 DEBUG awx.main.dispatch task 1d758e98-70ae-4db0-aa7b-a43fa4ff901d starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:34:15,709 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:34:15,735 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:34:15,781 DEBUG awx.main.dispatch task 1d758e98-70ae-4db0-aa7b-a43fa4ff901d is finished
2022-08-28 03:34:27,980 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(94f49b24-f841-4336-96f2-cd2ebd0a0e9e, queue=awx_private_queue)
[2022-08-28 03:34:27,980: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(94f49b24-f841-4336-96f2-cd2ebd0a0e9e, queue=awx_private_queue)
2022-08-28 03:34:27,997 DEBUG awx.main.dispatch delivered 94f49b24-f841-4336-96f2-cd2ebd0a0e9e to worker[207] qsize 0
2022-08-28 03:34:28,000 DEBUG awx.main.dispatch task 94f49b24-f841-4336-96f2-cd2ebd0a0e9e starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:34:28,026 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:34:28,039 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:33:58.030474+00:00
2022-08-28 03:34:28,094 DEBUG awx.main.dispatch task 94f49b24-f841-4336-96f2-cd2ebd0a0e9e is finished
2022-08-28 03:34:35,475 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7e0c020a-606d-46d0-8ccf-9b24382cf668, queue=awx_private_queue)
[2022-08-28 03:34:35,475: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7e0c020a-606d-46d0-8ccf-9b24382cf668, queue=awx_private_queue)
2022-08-28 03:34:35,498 DEBUG awx.main.dispatch delivered 7e0c020a-606d-46d0-8ccf-9b24382cf668 to worker[199] qsize 0
2022-08-28 03:34:35,500 DEBUG awx.main.dispatch task 7e0c020a-606d-46d0-8ccf-9b24382cf668 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:34:35,503 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:34:35,528 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:34:35,562 DEBUG awx.main.dispatch task 7e0c020a-606d-46d0-8ccf-9b24382cf668 is finished
2022-08-28 03:34:55,479 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(03f96b2f-cc19-4e38-9bcf-a1dfc03a608e, queue=awx_private_queue)
[2022-08-28 03:34:55,479: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(03f96b2f-cc19-4e38-9bcf-a1dfc03a608e, queue=awx_private_queue)
2022-08-28 03:34:55,499 DEBUG awx.main.dispatch delivered 03f96b2f-cc19-4e38-9bcf-a1dfc03a608e to worker[199] qsize 0
2022-08-28 03:34:55,503 DEBUG awx.main.dispatch task 03f96b2f-cc19-4e38-9bcf-a1dfc03a608e starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:34:55,505 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:34:55,532 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:34:55,585 DEBUG awx.main.dispatch task 03f96b2f-cc19-4e38-9bcf-a1dfc03a608e is finished
2022-08-28 03:34:57,981 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6e02dfd6-e7cd-4a0a-a5e3-a1389cfbde60, queue=awx_private_queue)
[2022-08-28 03:34:57,981: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6e02dfd6-e7cd-4a0a-a5e3-a1389cfbde60, queue=awx_private_queue)
2022-08-28 03:34:57,999 DEBUG awx.main.dispatch delivered 6e02dfd6-e7cd-4a0a-a5e3-a1389cfbde60 to worker[202] qsize 0
2022-08-28 03:34:58,002 DEBUG awx.main.dispatch task 6e02dfd6-e7cd-4a0a-a5e3-a1389cfbde60 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:34:58,030 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:34:58,050 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:34:28.026566+00:00
2022-08-28 03:34:58,079 DEBUG awx.main.dispatch task 6e02dfd6-e7cd-4a0a-a5e3-a1389cfbde60 is finished
RESULT 2
OKREADY
2022-08-28 03:35:08,956 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(4851cfbf-f7c7-497f-b6b7-ba58d81d9e6b, queue=onap-awx-0)
[2022-08-28 03:35:08,956: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(4851cfbf-f7c7-497f-b6b7-ba58d81d9e6b, queue=onap-awx-0)
2022-08-28 03:35:09,002 DEBUG awx.main.dispatch delivered 4851cfbf-f7c7-497f-b6b7-ba58d81d9e6b to worker[207] qsize 0
2022-08-28 03:35:09,005 DEBUG awx.main.dispatch task 4851cfbf-f7c7-497f-b6b7-ba58d81d9e6b starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:35:09,008 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:35:09,048 DEBUG awx.main.dispatch task 4851cfbf-f7c7-497f-b6b7-ba58d81d9e6b is finished
2022-08-28 03:35:15,213 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ba15ce45-4855-4581-9f51-68d7595fd391, queue=onap-awx-0)
[2022-08-28 03:35:15,213: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ba15ce45-4855-4581-9f51-68d7595fd391, queue=onap-awx-0)
2022-08-28 03:35:15,284 DEBUG awx.main.dispatch delivered ba15ce45-4855-4581-9f51-68d7595fd391 to worker[199] qsize 0
2022-08-28 03:35:15,286 DEBUG awx.main.dispatch task ba15ce45-4855-4581-9f51-68d7595fd391 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:35:15,307 DEBUG awx.main.dispatch task ba15ce45-4855-4581-9f51-68d7595fd391 is finished
2022-08-28 03:35:15,478 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0f08518f-fd7b-4508-856e-60ba81f2e47d, queue=awx_private_queue)
[2022-08-28 03:35:15,478: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0f08518f-fd7b-4508-856e-60ba81f2e47d, queue=awx_private_queue)
2022-08-28 03:35:15,693 DEBUG awx.main.dispatch delivered 0f08518f-fd7b-4508-856e-60ba81f2e47d to worker[209] qsize 0
2022-08-28 03:35:15,696 DEBUG awx.main.dispatch task 0f08518f-fd7b-4508-856e-60ba81f2e47d starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:35:15,698 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:35:15,722 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:35:15,802 DEBUG awx.main.dispatch task 0f08518f-fd7b-4508-856e-60ba81f2e47d is finished
2022-08-28 03:35:27,984 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(32f1d433-f255-4651-b22a-748b1ee23406, queue=awx_private_queue)
[2022-08-28 03:35:27,984: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(32f1d433-f255-4651-b22a-748b1ee23406, queue=awx_private_queue)
2022-08-28 03:35:28,000 DEBUG awx.main.dispatch delivered 32f1d433-f255-4651-b22a-748b1ee23406 to worker[209] qsize 0
2022-08-28 03:35:28,004 DEBUG awx.main.dispatch task 32f1d433-f255-4651-b22a-748b1ee23406 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:35:28,045 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:35:28,053 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:34:58.030954+00:00
2022-08-28 03:35:28,076 DEBUG awx.main.dispatch task 32f1d433-f255-4651-b22a-748b1ee23406 is finished
2022-08-28 03:35:35,478 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(89ee9b90-fdb5-4ea3-8295-fe3733a71a9f, queue=awx_private_queue)
[2022-08-28 03:35:35,478: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(89ee9b90-fdb5-4ea3-8295-fe3733a71a9f, queue=awx_private_queue)
2022-08-28 03:35:35,494 DEBUG awx.main.dispatch delivered 89ee9b90-fdb5-4ea3-8295-fe3733a71a9f to worker[202] qsize 0
2022-08-28 03:35:35,498 DEBUG awx.main.dispatch task 89ee9b90-fdb5-4ea3-8295-fe3733a71a9f starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:35:35,500 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:35:35,522 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:35:35,585 DEBUG awx.main.dispatch task 89ee9b90-fdb5-4ea3-8295-fe3733a71a9f is finished
2022-08-28 03:35:55,478 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c72cb619-70e3-4d20-942d-b62dd7e7b0ce, queue=awx_private_queue)
[2022-08-28 03:35:55,478: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c72cb619-70e3-4d20-942d-b62dd7e7b0ce, queue=awx_private_queue)
2022-08-28 03:35:55,497 DEBUG awx.main.dispatch delivered c72cb619-70e3-4d20-942d-b62dd7e7b0ce to worker[199] qsize 0
2022-08-28 03:35:55,500 DEBUG awx.main.dispatch task c72cb619-70e3-4d20-942d-b62dd7e7b0ce starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:35:55,503 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:35:55,526 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:35:55,572 DEBUG awx.main.dispatch task c72cb619-70e3-4d20-942d-b62dd7e7b0ce is finished
2022-08-28 03:35:57,983 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c9a6bfeb-7ffd-484d-9e82-f86853f6f2ba, queue=awx_private_queue)
[2022-08-28 03:35:57,983: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c9a6bfeb-7ffd-484d-9e82-f86853f6f2ba, queue=awx_private_queue)
2022-08-28 03:35:58,014 DEBUG awx.main.dispatch delivered c9a6bfeb-7ffd-484d-9e82-f86853f6f2ba to worker[199] qsize 0
2022-08-28 03:35:58,017 DEBUG awx.main.dispatch task c9a6bfeb-7ffd-484d-9e82-f86853f6f2ba starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:35:58,030 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:35:58,034 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:35:28.045247+00:00
2022-08-28 03:35:58,052 DEBUG awx.main.dispatch task c9a6bfeb-7ffd-484d-9e82-f86853f6f2ba is finished
RESULT 2
OKREADY
2022-08-28 03:36:08,958 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(2fa1fb64-7a9f-474e-a10a-7723fc595708, queue=onap-awx-0)
[2022-08-28 03:36:08,958: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(2fa1fb64-7a9f-474e-a10a-7723fc595708, queue=onap-awx-0)
2022-08-28 03:36:08,988 DEBUG awx.main.dispatch delivered 2fa1fb64-7a9f-474e-a10a-7723fc595708 to worker[209] qsize 0
2022-08-28 03:36:08,990 DEBUG awx.main.dispatch task 2fa1fb64-7a9f-474e-a10a-7723fc595708 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:36:08,992 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:36:09,027 DEBUG awx.main.dispatch task 2fa1fb64-7a9f-474e-a10a-7723fc595708 is finished
2022-08-28 03:36:15,211 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b7d75448-9e12-4fb5-8a0e-53f8bdf6aefa, queue=onap-awx-0)
[2022-08-28 03:36:15,211: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b7d75448-9e12-4fb5-8a0e-53f8bdf6aefa, queue=onap-awx-0)
2022-08-28 03:36:15,249 DEBUG awx.main.dispatch delivered b7d75448-9e12-4fb5-8a0e-53f8bdf6aefa to worker[209] qsize 0
2022-08-28 03:36:15,251 DEBUG awx.main.dispatch task b7d75448-9e12-4fb5-8a0e-53f8bdf6aefa starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:36:15,268 DEBUG awx.main.dispatch task b7d75448-9e12-4fb5-8a0e-53f8bdf6aefa is finished
2022-08-28 03:36:15,478 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(014a278d-440f-49b0-b4d0-3c8752977d7d, queue=awx_private_queue)
[2022-08-28 03:36:15,478: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(014a278d-440f-49b0-b4d0-3c8752977d7d, queue=awx_private_queue)
2022-08-28 03:36:15,688 DEBUG awx.main.dispatch delivered 014a278d-440f-49b0-b4d0-3c8752977d7d to worker[199] qsize 0
2022-08-28 03:36:15,690 DEBUG awx.main.dispatch task 014a278d-440f-49b0-b4d0-3c8752977d7d starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:36:15,692 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:36:15,711 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:36:15,753 DEBUG awx.main.dispatch task 014a278d-440f-49b0-b4d0-3c8752977d7d is finished
2022-08-28 03:36:27,986 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6e45aa42-35c3-47c5-aa85-e5dd7925a679, queue=awx_private_queue)
[2022-08-28 03:36:27,986: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6e45aa42-35c3-47c5-aa85-e5dd7925a679, queue=awx_private_queue)
2022-08-28 03:36:28,015 DEBUG awx.main.dispatch delivered 6e45aa42-35c3-47c5-aa85-e5dd7925a679 to worker[207] qsize 0
2022-08-28 03:36:28,018 DEBUG awx.main.dispatch task 6e45aa42-35c3-47c5-aa85-e5dd7925a679 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:36:28,035 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:36:28,038 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:35:58.031098+00:00
2022-08-28 03:36:28,057 DEBUG awx.main.dispatch task 6e45aa42-35c3-47c5-aa85-e5dd7925a679 is finished
2022-08-28 03:36:35,478 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(28903da7-ec59-4378-9520-45e4f3ffe56f, queue=awx_private_queue)
[2022-08-28 03:36:35,478: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(28903da7-ec59-4378-9520-45e4f3ffe56f, queue=awx_private_queue)
2022-08-28 03:36:35,494 DEBUG awx.main.dispatch delivered 28903da7-ec59-4378-9520-45e4f3ffe56f to worker[207] qsize 0
2022-08-28 03:36:35,497 DEBUG awx.main.dispatch task 28903da7-ec59-4378-9520-45e4f3ffe56f starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:36:35,498 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:36:35,522 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:36:35,562 DEBUG awx.main.dispatch task 28903da7-ec59-4378-9520-45e4f3ffe56f is finished
2022-08-28 03:36:55,489 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a1b2da58-ac3c-40f7-83e3-6ca45105bef3, queue=awx_private_queue)
[2022-08-28 03:36:55,489: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a1b2da58-ac3c-40f7-83e3-6ca45105bef3, queue=awx_private_queue)
2022-08-28 03:36:55,513 DEBUG awx.main.dispatch delivered a1b2da58-ac3c-40f7-83e3-6ca45105bef3 to worker[209] qsize 0
2022-08-28 03:36:55,515 DEBUG awx.main.dispatch task a1b2da58-ac3c-40f7-83e3-6ca45105bef3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:36:55,518 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:36:55,544 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:36:55,604 DEBUG awx.main.dispatch task a1b2da58-ac3c-40f7-83e3-6ca45105bef3 is finished
2022-08-28 03:36:57,986 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(35e7ddd2-ec63-4091-b369-a6fe90fad3ec, queue=awx_private_queue)
[2022-08-28 03:36:57,986: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(35e7ddd2-ec63-4091-b369-a6fe90fad3ec, queue=awx_private_queue)
2022-08-28 03:36:58,003 DEBUG awx.main.dispatch delivered 35e7ddd2-ec63-4091-b369-a6fe90fad3ec to worker[202] qsize 0
2022-08-28 03:36:58,005 DEBUG awx.main.dispatch task 35e7ddd2-ec63-4091-b369-a6fe90fad3ec starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:36:58,037 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:36:58,041 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:36:28.035204+00:00
2022-08-28 03:36:58,102 DEBUG awx.main.dispatch task 35e7ddd2-ec63-4091-b369-a6fe90fad3ec is finished
RESULT 2
OKREADY
2022-08-28 03:37:08,959 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(1a774f47-a39c-4a55-979e-1432838e9ba6, queue=onap-awx-0)
[2022-08-28 03:37:08,959: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(1a774f47-a39c-4a55-979e-1432838e9ba6, queue=onap-awx-0)
2022-08-28 03:37:09,012 DEBUG awx.main.dispatch delivered 1a774f47-a39c-4a55-979e-1432838e9ba6 to worker[202] qsize 0
2022-08-28 03:37:09,014 DEBUG awx.main.dispatch task 1a774f47-a39c-4a55-979e-1432838e9ba6 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:37:09,015 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:37:09,153 DEBUG awx.main.dispatch task 1a774f47-a39c-4a55-979e-1432838e9ba6 is finished
2022-08-28 03:37:15,211 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(913c076a-a005-4a73-a6b3-4c799e8e5f2d, queue=onap-awx-0)
[2022-08-28 03:37:15,211: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(913c076a-a005-4a73-a6b3-4c799e8e5f2d, queue=onap-awx-0)
2022-08-28 03:37:15,226 DEBUG awx.main.dispatch delivered 913c076a-a005-4a73-a6b3-4c799e8e5f2d to worker[209] qsize 0
2022-08-28 03:37:15,239 DEBUG awx.main.dispatch task 913c076a-a005-4a73-a6b3-4c799e8e5f2d starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:37:15,257 DEBUG awx.main.dispatch task 913c076a-a005-4a73-a6b3-4c799e8e5f2d is finished
2022-08-28 03:37:15,488 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(682e34f2-2fac-43b1-a961-b1a25fddcee1, queue=awx_private_queue)
[2022-08-28 03:37:15,488: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(682e34f2-2fac-43b1-a961-b1a25fddcee1, queue=awx_private_queue)
2022-08-28 03:37:15,705 DEBUG awx.main.dispatch delivered 682e34f2-2fac-43b1-a961-b1a25fddcee1 to worker[202] qsize 0
2022-08-28 03:37:15,707 DEBUG awx.main.dispatch task 682e34f2-2fac-43b1-a961-b1a25fddcee1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:37:15,708 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:37:15,747 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:37:15,788 DEBUG awx.main.dispatch task 682e34f2-2fac-43b1-a961-b1a25fddcee1 is finished
2022-08-28 03:37:27,987 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dcea9974-4fbc-4c8f-a7fd-fcd863d57630, queue=awx_private_queue)
[2022-08-28 03:37:27,987: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dcea9974-4fbc-4c8f-a7fd-fcd863d57630, queue=awx_private_queue)
2022-08-28 03:37:28,008 DEBUG awx.main.dispatch delivered dcea9974-4fbc-4c8f-a7fd-fcd863d57630 to worker[207] qsize 0
2022-08-28 03:37:28,011 DEBUG awx.main.dispatch task dcea9974-4fbc-4c8f-a7fd-fcd863d57630 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:37:28,040 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:37:28,046 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:36:58.037286+00:00
2022-08-28 03:37:28,080 DEBUG awx.main.dispatch task dcea9974-4fbc-4c8f-a7fd-fcd863d57630 is finished
2022-08-28 03:37:35,489 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d5c753f4-3945-43ff-a17a-9bfa4e66ad85, queue=awx_private_queue)
[2022-08-28 03:37:35,489: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d5c753f4-3945-43ff-a17a-9bfa4e66ad85, queue=awx_private_queue)
2022-08-28 03:37:35,507 DEBUG awx.main.dispatch delivered d5c753f4-3945-43ff-a17a-9bfa4e66ad85 to worker[207] qsize 0
2022-08-28 03:37:35,511 DEBUG awx.main.dispatch task d5c753f4-3945-43ff-a17a-9bfa4e66ad85 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:37:35,512 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:37:35,535 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:37:35,589 DEBUG awx.main.dispatch task d5c753f4-3945-43ff-a17a-9bfa4e66ad85 is finished
2022-08-28 03:37:55,495 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(59009c4f-9dc7-479b-b37c-24513381a59b, queue=awx_private_queue)
[2022-08-28 03:37:55,495: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(59009c4f-9dc7-479b-b37c-24513381a59b, queue=awx_private_queue)
2022-08-28 03:37:55,512 DEBUG awx.main.dispatch delivered 59009c4f-9dc7-479b-b37c-24513381a59b to worker[209] qsize 0
2022-08-28 03:37:55,515 DEBUG awx.main.dispatch task 59009c4f-9dc7-479b-b37c-24513381a59b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:37:55,517 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:37:55,550 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:37:55,585 DEBUG awx.main.dispatch task 59009c4f-9dc7-479b-b37c-24513381a59b is finished
2022-08-28 03:37:57,987 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5d4b314c-7eb7-4937-b044-ae67c59112fb, queue=awx_private_queue)
[2022-08-28 03:37:57,987: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5d4b314c-7eb7-4937-b044-ae67c59112fb, queue=awx_private_queue)
2022-08-28 03:37:58,002 DEBUG awx.main.dispatch delivered 5d4b314c-7eb7-4937-b044-ae67c59112fb to worker[207] qsize 0
2022-08-28 03:37:58,004 DEBUG awx.main.dispatch task 5d4b314c-7eb7-4937-b044-ae67c59112fb starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:37:58,037 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:37:58,050 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:37:28.040647+00:00
2022-08-28 03:37:58,069 DEBUG awx.main.dispatch task 5d4b314c-7eb7-4937-b044-ae67c59112fb is finished
RESULT 2
OKREADY
2022-08-28 03:38:08,960 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e9c75a36-67c6-4fd7-b46e-59bfe9cb9456, queue=onap-awx-0)
[2022-08-28 03:38:08,960: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e9c75a36-67c6-4fd7-b46e-59bfe9cb9456, queue=onap-awx-0)
2022-08-28 03:38:08,991 DEBUG awx.main.dispatch delivered e9c75a36-67c6-4fd7-b46e-59bfe9cb9456 to worker[199] qsize 0
2022-08-28 03:38:08,993 DEBUG awx.main.dispatch task e9c75a36-67c6-4fd7-b46e-59bfe9cb9456 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:38:08,995 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:38:09,033 DEBUG awx.main.dispatch task e9c75a36-67c6-4fd7-b46e-59bfe9cb9456 is finished
2022-08-28 03:38:15,211 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d2f4b0d9-a962-451d-8cbd-dbc141b1266e, queue=onap-awx-0)
[2022-08-28 03:38:15,211: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d2f4b0d9-a962-451d-8cbd-dbc141b1266e, queue=onap-awx-0)
2022-08-28 03:38:15,228 DEBUG awx.main.dispatch delivered d2f4b0d9-a962-451d-8cbd-dbc141b1266e to worker[202] qsize 0
2022-08-28 03:38:15,232 DEBUG awx.main.dispatch task d2f4b0d9-a962-451d-8cbd-dbc141b1266e starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:38:15,250 DEBUG awx.main.dispatch task d2f4b0d9-a962-451d-8cbd-dbc141b1266e is finished
2022-08-28 03:38:15,495 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8a1ee2df-383f-48d0-9994-2b82a44117e1, queue=awx_private_queue)
[2022-08-28 03:38:15,495: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8a1ee2df-383f-48d0-9994-2b82a44117e1, queue=awx_private_queue)
2022-08-28 03:38:15,712 DEBUG awx.main.dispatch delivered 8a1ee2df-383f-48d0-9994-2b82a44117e1 to worker[207] qsize 0
2022-08-28 03:38:15,715 DEBUG awx.main.dispatch task 8a1ee2df-383f-48d0-9994-2b82a44117e1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:38:15,717 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:38:15,748 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:38:15,829 DEBUG awx.main.dispatch task 8a1ee2df-383f-48d0-9994-2b82a44117e1 is finished
2022-08-28 03:38:27,990 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ecee522b-387c-4ee6-aef5-b676be4c4b2e, queue=awx_private_queue)
[2022-08-28 03:38:27,990: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ecee522b-387c-4ee6-aef5-b676be4c4b2e, queue=awx_private_queue)
2022-08-28 03:38:28,008 DEBUG awx.main.dispatch delivered ecee522b-387c-4ee6-aef5-b676be4c4b2e to worker[202] qsize 0
2022-08-28 03:38:28,010 DEBUG awx.main.dispatch task ecee522b-387c-4ee6-aef5-b676be4c4b2e starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:38:28,055 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:38:28,061 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:37:58.038181+00:00
2022-08-28 03:38:28,147 DEBUG awx.main.dispatch task ecee522b-387c-4ee6-aef5-b676be4c4b2e is finished
2022-08-28 03:38:35,495 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e5fb596a-9cfa-4b09-8148-a21264a72e41, queue=awx_private_queue)
[2022-08-28 03:38:35,495: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e5fb596a-9cfa-4b09-8148-a21264a72e41, queue=awx_private_queue)
2022-08-28 03:38:35,511 DEBUG awx.main.dispatch delivered e5fb596a-9cfa-4b09-8148-a21264a72e41 to worker[207] qsize 0
2022-08-28 03:38:35,514 DEBUG awx.main.dispatch task e5fb596a-9cfa-4b09-8148-a21264a72e41 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:38:35,517 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:38:35,543 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:38:35,598 DEBUG awx.main.dispatch task e5fb596a-9cfa-4b09-8148-a21264a72e41 is finished
2022-08-28 03:38:55,498 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(94953f49-7b3d-491b-a978-051112c6bab8, queue=awx_private_queue)
[2022-08-28 03:38:55,498: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(94953f49-7b3d-491b-a978-051112c6bab8, queue=awx_private_queue)
2022-08-28 03:38:55,522 DEBUG awx.main.dispatch delivered 94953f49-7b3d-491b-a978-051112c6bab8 to worker[207] qsize 0
2022-08-28 03:38:55,526 DEBUG awx.main.dispatch task 94953f49-7b3d-491b-a978-051112c6bab8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:38:55,528 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:38:55,550 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:38:55,587 DEBUG awx.main.dispatch task 94953f49-7b3d-491b-a978-051112c6bab8 is finished
2022-08-28 03:38:57,990 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8734e90d-737c-42a5-b606-12ff055ba886, queue=awx_private_queue)
[2022-08-28 03:38:57,990: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8734e90d-737c-42a5-b606-12ff055ba886, queue=awx_private_queue)
2022-08-28 03:38:58,006 DEBUG awx.main.dispatch delivered 8734e90d-737c-42a5-b606-12ff055ba886 to worker[209] qsize 0
2022-08-28 03:38:58,010 DEBUG awx.main.dispatch task 8734e90d-737c-42a5-b606-12ff055ba886 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:38:58,037 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:38:58,045 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:38:28.055860+00:00
2022-08-28 03:38:58,096 DEBUG awx.main.dispatch task 8734e90d-737c-42a5-b606-12ff055ba886 is finished
RESULT 2
OKREADY
2022-08-28 03:39:08,962 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(79d79ca7-fc67-46fa-ac5d-33b4c886ce72, queue=onap-awx-0)
[2022-08-28 03:39:08,962: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(79d79ca7-fc67-46fa-ac5d-33b4c886ce72, queue=onap-awx-0)
2022-08-28 03:39:09,005 DEBUG awx.main.dispatch delivered 79d79ca7-fc67-46fa-ac5d-33b4c886ce72 to worker[202] qsize 0
2022-08-28 03:39:09,008 DEBUG awx.main.dispatch task 79d79ca7-fc67-46fa-ac5d-33b4c886ce72 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:39:09,010 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:39:09,107 DEBUG awx.main.dispatch task 79d79ca7-fc67-46fa-ac5d-33b4c886ce72 is finished
2022-08-28 03:39:15,211 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(fae77413-7ea6-4ee9-ba59-35ebee17c66b, queue=onap-awx-0)
[2022-08-28 03:39:15,211: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(fae77413-7ea6-4ee9-ba59-35ebee17c66b, queue=onap-awx-0)
2022-08-28 03:39:15,226 DEBUG awx.main.dispatch delivered fae77413-7ea6-4ee9-ba59-35ebee17c66b to worker[202] qsize 0
2022-08-28 03:39:15,229 DEBUG awx.main.dispatch task fae77413-7ea6-4ee9-ba59-35ebee17c66b starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:39:15,248 DEBUG awx.main.dispatch task fae77413-7ea6-4ee9-ba59-35ebee17c66b is finished
2022-08-28 03:39:15,497 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(28660b27-abdd-4bd6-a443-742782b007a4, queue=awx_private_queue)
[2022-08-28 03:39:15,497: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(28660b27-abdd-4bd6-a443-742782b007a4, queue=awx_private_queue)
2022-08-28 03:39:15,714 DEBUG awx.main.dispatch delivered 28660b27-abdd-4bd6-a443-742782b007a4 to worker[209] qsize 0
2022-08-28 03:39:15,717 DEBUG awx.main.dispatch task 28660b27-abdd-4bd6-a443-742782b007a4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:39:15,719 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:39:15,767 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:39:15,880 DEBUG awx.main.dispatch task 28660b27-abdd-4bd6-a443-742782b007a4 is finished
2022-08-28 03:39:27,996 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6413952a-c649-4dfb-aafe-d8f82fcac8c4, queue=awx_private_queue)
[2022-08-28 03:39:27,996: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6413952a-c649-4dfb-aafe-d8f82fcac8c4, queue=awx_private_queue)
2022-08-28 03:39:28,012 DEBUG awx.main.dispatch delivered 6413952a-c649-4dfb-aafe-d8f82fcac8c4 to worker[199] qsize 0
2022-08-28 03:39:28,015 DEBUG awx.main.dispatch task 6413952a-c649-4dfb-aafe-d8f82fcac8c4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:39:28,052 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:39:28,073 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:38:58.037796+00:00
2022-08-28 03:39:28,115 DEBUG awx.main.dispatch task 6413952a-c649-4dfb-aafe-d8f82fcac8c4 is finished
2022-08-28 03:39:35,497 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e4e33837-6d52-48aa-93ba-d1a2af8e74c6, queue=awx_private_queue)
[2022-08-28 03:39:35,497: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e4e33837-6d52-48aa-93ba-d1a2af8e74c6, queue=awx_private_queue)
2022-08-28 03:39:35,513 DEBUG awx.main.dispatch delivered e4e33837-6d52-48aa-93ba-d1a2af8e74c6 to worker[207] qsize 0
2022-08-28 03:39:35,517 DEBUG awx.main.dispatch task e4e33837-6d52-48aa-93ba-d1a2af8e74c6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:39:35,519 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:39:35,544 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:39:35,581 DEBUG awx.main.dispatch task e4e33837-6d52-48aa-93ba-d1a2af8e74c6 is finished
2022-08-28 03:39:55,503 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6db4b1dd-5070-44ae-9670-5f04c78b3037, queue=awx_private_queue)
[2022-08-28 03:39:55,503: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6db4b1dd-5070-44ae-9670-5f04c78b3037, queue=awx_private_queue)
2022-08-28 03:39:55,518 DEBUG awx.main.dispatch delivered 6db4b1dd-5070-44ae-9670-5f04c78b3037 to worker[199] qsize 0
2022-08-28 03:39:55,520 DEBUG awx.main.dispatch task 6db4b1dd-5070-44ae-9670-5f04c78b3037 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:39:55,521 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:39:55,549 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:39:55,588 DEBUG awx.main.dispatch task 6db4b1dd-5070-44ae-9670-5f04c78b3037 is finished
2022-08-28 03:39:57,995 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d7db426e-5c0c-4286-8957-e886d4b80196, queue=awx_private_queue)
[2022-08-28 03:39:57,995: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d7db426e-5c0c-4286-8957-e886d4b80196, queue=awx_private_queue)
2022-08-28 03:39:58,019 DEBUG awx.main.dispatch delivered d7db426e-5c0c-4286-8957-e886d4b80196 to worker[209] qsize 0
2022-08-28 03:39:58,022 DEBUG awx.main.dispatch task d7db426e-5c0c-4286-8957-e886d4b80196 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-08-28 03:39:58,049 DEBUG awx.main.tasks Starting periodic scheduler
2022-08-28 03:39:58,054 DEBUG awx.main.tasks Last scheduler run was: 2022-08-28 03:39:28.052482+00:00
2022-08-28 03:39:58,084 DEBUG awx.main.dispatch task d7db426e-5c0c-4286-8957-e886d4b80196 is finished
RESULT 2
OKREADY
2022-08-28 03:40:08,963 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d7a0c674-e108-42b2-a579-17d8a9784543, queue=onap-awx-0)
[2022-08-28 03:40:08,963: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d7a0c674-e108-42b2-a579-17d8a9784543, queue=onap-awx-0)
2022-08-28 03:40:09,004 DEBUG awx.main.dispatch delivered d7a0c674-e108-42b2-a579-17d8a9784543 to worker[209] qsize 0
2022-08-28 03:40:09,006 DEBUG awx.main.dispatch task d7a0c674-e108-42b2-a579-17d8a9784543 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-08-28 03:40:09,008 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-08-28 03:40:09,052 DEBUG awx.main.dispatch task d7a0c674-e108-42b2-a579-17d8a9784543 is finished
2022-08-28 03:40:15,211 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d42d2f7e-8a90-4ff5-9367-7b0a568e3196, queue=onap-awx-0)
[2022-08-28 03:40:15,211: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d42d2f7e-8a90-4ff5-9367-7b0a568e3196, queue=onap-awx-0)
2022-08-28 03:40:15,227 DEBUG awx.main.dispatch delivered d42d2f7e-8a90-4ff5-9367-7b0a568e3196 to worker[209] qsize 0
2022-08-28 03:40:15,229 DEBUG awx.main.dispatch task d42d2f7e-8a90-4ff5-9367-7b0a568e3196 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-08-28 03:40:15,247 DEBUG awx.main.dispatch task d42d2f7e-8a90-4ff5-9367-7b0a568e3196 is finished
2022-08-28 03:40:15,503 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d98e4013-dcb5-42f3-a976-c53860b90100, queue=awx_private_queue)
[2022-08-28 03:40:15,503: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d98e4013-dcb5-42f3-a976-c53860b90100, queue=awx_private_queue)
2022-08-28 03:40:15,718 DEBUG awx.main.dispatch delivered d98e4013-dcb5-42f3-a976-c53860b90100 to worker[207] qsize 0
2022-08-28 03:40:15,722 DEBUG awx.main.dispatch task d98e4013-dcb5-42f3-a976-c53860b90100 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-08-28 03:40:15,723 DEBUG awx.main.scheduler Running Tower task manager.
2022-08-28 03:40:15,785 DEBUG awx.main.scheduler Starting Scheduler
2022-08-28 03:40:15,870 DEBUG awx.main.dispatch task d98e4013-dcb5-42f3-a976-c53860b90100 is finished
2022-08-28 03:40:27,996 DEBUG awx.main.dispatch publish awx.main.tasks.awx