By type
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"elapsed": 13,
"match_groupdict": {},
"match_groups": [],
"path": null,
"port": 5432,
"search_regex": null,
"state": "started"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | CHANGED => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": true,
"db": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
}
Default organization added.
Demo Credential, Inventory, and Job Template added.
(changed: True)
2022-07-25 23:16:34,575 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(65dbd7bb-6a82-4dbc-b00e-d3df345dc0c2, queue=awx_private_queue)
Successfully registered instance onap-awx-0
(changed: True)
2022-07-25 23:16:39,640 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(1124ec97-618f-4811-a3fd-e2dd1842bb13, queue=awx_private_queue)
2022-07-25 23:16:39,719 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(48da838e-1368-41f9-a36a-bf34f6eb40de, queue=awx_private_queue)
Creating instance group tower
(changed: True)
/usr/bin/launch_awx_task.sh: line 27: unset: `#!/bin/sh': not a valid identifier
2022-07-25 23:16:40,674 INFO RPC interface 'supervisor' initialized
2022-07-25 23:16:40,674 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-07-25 23:16:40,674 INFO supervisord started with pid 150
2022-07-25 23:16:41,679 INFO spawned: 'awx-config-watcher' with pid 153
2022-07-25 23:16:41,684 INFO spawned: 'dispatcher' with pid 154
2022-07-25 23:16:41,688 INFO spawned: 'callback-receiver' with pid 155
2022-07-25 23:16:41,693 INFO spawned: 'channels-worker' with pid 156
READY
2022-07-25 23:16:42,838 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-07-25 23:16:42,838 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-07-25 23:16:42,839 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-07-25 23:16:42,839 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-07-25 23:16:46,114 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2022-07-25 23:16:46,114 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2022-07-25 23:16:46,123 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2022-07-25 23:16:46,123 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2022-07-25 23:16:46,130 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2022-07-25 23:16:46,130 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2022-07-25 23:16:46,138 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2022-07-25 23:16:46,138 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2022-07-25 23:16:46,147 - INFO - runworker - Using single-threaded worker.
2022-07-25 23:16:46,148 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer)
2022-07-25 23:16:46,149 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive
2022-07-25 23:16:46,165 ERROR celery.beat Removing corrupted schedule file '/var/lib/awx/beat.db': error(11, 'Resource temporarily unavailable')
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 485, in setup_schedule
self._store = self._open_schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 475, in _open_schedule
return self.persistence.open(self.schedule_filename, writeback=True)
File "/usr/lib64/python3.6/shelve.py", line 243, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/usr/lib64/python3.6/shelve.py", line 227, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
File "/usr/lib64/python3.6/dbm/__init__.py", line 94, in open
return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable
2022-07-25 23:16:46,206 WARNING awx.main.dispatch scaling up worker pid:180
2022-07-25 23:16:46,225 WARNING awx.main.dispatch scaling up worker pid:181
2022-07-25 23:16:46,240 WARNING awx.main.dispatch scaling up worker pid:182
2022-07-25 23:16:46,255 WARNING awx.main.dispatch scaling up worker pid:183
2022-07-25 23:16:46,264 DEBUG awx.main.tasks Syncing Schedules
2022-07-25 23:16:46,600 DEBUG awx.main.tasks Waited 0.006929159164428711 seconds to obtain lock name: cluster_policy_lock
2022-07-25 23:16:46,626 DEBUG awx.main.tasks Total non-isolated instances:1 available for policy: 1
2022-07-25 23:16:46,627 DEBUG awx.main.tasks Policy percentage, adding Instances [1] to Group tower
2022-07-25 23:16:46,627 DEBUG awx.main.tasks Adding instances [1] to group tower
2022-07-25 23:16:46,649 DEBUG awx.main.tasks Cluster policy computation finished in 0.04808449745178223 seconds
2022-07-25 23:16:46,650 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:16:46,734 DEBUG awx.main.dispatch listening on ['awx_private_queue [direct]', 'onap-awx-0 [direct]', 'tower_broadcast_all_onap-awx-0 [fanout]']
RESULT 2
OKREADY
2022-07-25 23:17:06,398 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(db984cb4-511e-4cab-8d4d-e768cdc336fd, queue=awx_private_queue)
[2022-07-25 23:17:06,398: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(db984cb4-511e-4cab-8d4d-e768cdc336fd, queue=awx_private_queue)
2022-07-25 23:17:06,419 DEBUG awx.main.dispatch delivered db984cb4-511e-4cab-8d4d-e768cdc336fd to worker[181] qsize 0
2022-07-25 23:17:06,426 DEBUG awx.main.dispatch task db984cb4-511e-4cab-8d4d-e768cdc336fd starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:17:06,430 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:17:06,475 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:17:06,559 DEBUG awx.main.dispatch task db984cb4-511e-4cab-8d4d-e768cdc336fd is finished
2022-07-25 23:17:16,380 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e740f680-d816-4376-b143-3038efc89636, queue=awx_private_queue)
[2022-07-25 23:17:16,380: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e740f680-d816-4376-b143-3038efc89636, queue=awx_private_queue)
2022-07-25 23:17:16,398 DEBUG awx.main.dispatch delivered e740f680-d816-4376-b143-3038efc89636 to worker[181] qsize 0
2022-07-25 23:17:16,400 DEBUG awx.main.dispatch task e740f680-d816-4376-b143-3038efc89636 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:17:16,422 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:17:16,447 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:17:16.430726+00:00
2022-07-25 23:17:16,499 DEBUG awx.main.dispatch task e740f680-d816-4376-b143-3038efc89636 is finished
2022-07-25 23:17:26,380 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cc509026-2063-44ad-9860-daca6c403f65, queue=awx_private_queue)
[2022-07-25 23:17:26,380: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cc509026-2063-44ad-9860-daca6c403f65, queue=awx_private_queue)
2022-07-25 23:17:26,449 DEBUG awx.main.dispatch delivered cc509026-2063-44ad-9860-daca6c403f65 to worker[183] qsize 0
2022-07-25 23:17:26,455 DEBUG awx.main.dispatch task cc509026-2063-44ad-9860-daca6c403f65 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:17:26,458 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:17:26,507 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:17:26,593 DEBUG awx.main.dispatch task cc509026-2063-44ad-9860-daca6c403f65 is finished
2022-07-25 23:17:46,389 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6a76159a-f9b6-4df1-8ca9-348497f420fa, queue=onap-awx-0)
[2022-07-25 23:17:46,389: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6a76159a-f9b6-4df1-8ca9-348497f420fa, queue=onap-awx-0)
2022-07-25 23:17:46,671 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a3051bca-9fbe-4acf-8d74-3d0715e5eb86, queue=onap-awx-0)
[2022-07-25 23:17:46,671: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a3051bca-9fbe-4acf-8d74-3d0715e5eb86, queue=onap-awx-0)
2022-07-25 23:17:46,691 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(399639b7-ca03-48dd-9b63-b6843dce383a, queue=awx_private_queue)
[2022-07-25 23:17:46,691: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(399639b7-ca03-48dd-9b63-b6843dce383a, queue=awx_private_queue)
2022-07-25 23:17:46,714 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8a46e31a-3516-4447-a65b-34e26de8d5c1, queue=awx_private_queue)
[2022-07-25 23:17:46,714: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8a46e31a-3516-4447-a65b-34e26de8d5c1, queue=awx_private_queue)
2022-07-25 23:17:46,716 DEBUG awx.main.dispatch delivered 6a76159a-f9b6-4df1-8ca9-348497f420fa to worker[180] qsize 0
2022-07-25 23:17:46,720 DEBUG awx.main.dispatch delivered a3051bca-9fbe-4acf-8d74-3d0715e5eb86 to worker[181] qsize 0
2022-07-25 23:17:46,722 DEBUG awx.main.dispatch delivered 399639b7-ca03-48dd-9b63-b6843dce383a to worker[182] qsize 0
2022-07-25 23:17:46,723 DEBUG awx.main.dispatch task a3051bca-9fbe-4acf-8d74-3d0715e5eb86 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:17:46,725 DEBUG awx.main.dispatch task 399639b7-ca03-48dd-9b63-b6843dce383a starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:17:46,721 DEBUG awx.main.dispatch task 6a76159a-f9b6-4df1-8ca9-348497f420fa starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:17:46,733 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:17:46,734 DEBUG awx.main.dispatch delivered 8a46e31a-3516-4447-a65b-34e26de8d5c1 to worker[183] qsize 0
2022-07-25 23:17:46,737 DEBUG awx.main.dispatch task 8a46e31a-3516-4447-a65b-34e26de8d5c1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:17:46,741 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:17:46,773 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:17:46,785 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:17:16.423224+00:00
2022-07-25 23:17:46,793 DEBUG awx.main.dispatch task a3051bca-9fbe-4acf-8d74-3d0715e5eb86 is finished
2022-07-25 23:17:46,852 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:17:46,949 DEBUG awx.main.dispatch task 6a76159a-f9b6-4df1-8ca9-348497f420fa is finished
2022-07-25 23:17:47,023 DEBUG awx.main.dispatch task 399639b7-ca03-48dd-9b63-b6843dce383a is finished
2022-07-25 23:17:47,055 DEBUG awx.main.dispatch task 8a46e31a-3516-4447-a65b-34e26de8d5c1 is finished
RESULT 2
OKREADY
2022-07-25 23:18:06,725 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8be236c2-20de-466d-9f20-372b44e9df9e, queue=awx_private_queue)
[2022-07-25 23:18:06,725: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8be236c2-20de-466d-9f20-372b44e9df9e, queue=awx_private_queue)
2022-07-25 23:18:06,745 DEBUG awx.main.dispatch delivered 8be236c2-20de-466d-9f20-372b44e9df9e to worker[183] qsize 0
2022-07-25 23:18:06,748 DEBUG awx.main.dispatch task 8be236c2-20de-466d-9f20-372b44e9df9e starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:18:06,750 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:18:06,879 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:18:07,104 DEBUG awx.main.dispatch task 8be236c2-20de-466d-9f20-372b44e9df9e is finished
2022-07-25 23:18:16,691 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9bc2a120-b59f-4933-8702-0d1da40ddbe3, queue=awx_private_queue)
[2022-07-25 23:18:16,691: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9bc2a120-b59f-4933-8702-0d1da40ddbe3, queue=awx_private_queue)
2022-07-25 23:18:16,708 DEBUG awx.main.dispatch delivered 9bc2a120-b59f-4933-8702-0d1da40ddbe3 to worker[183] qsize 0
2022-07-25 23:18:16,711 DEBUG awx.main.dispatch task 9bc2a120-b59f-4933-8702-0d1da40ddbe3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:18:16,755 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:18:16,763 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:17:46.774210+00:00
2022-07-25 23:18:16,861 DEBUG awx.main.dispatch task 9bc2a120-b59f-4933-8702-0d1da40ddbe3 is finished
2022-07-25 23:18:26,724 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(09e1e4c5-488e-4e00-b480-68818dbfc559, queue=awx_private_queue)
[2022-07-25 23:18:26,724: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(09e1e4c5-488e-4e00-b480-68818dbfc559, queue=awx_private_queue)
2022-07-25 23:18:26,742 DEBUG awx.main.dispatch delivered 09e1e4c5-488e-4e00-b480-68818dbfc559 to worker[183] qsize 0
2022-07-25 23:18:26,745 DEBUG awx.main.dispatch task 09e1e4c5-488e-4e00-b480-68818dbfc559 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:18:26,747 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:18:27,063 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:18:27,556 DEBUG awx.main.dispatch task 09e1e4c5-488e-4e00-b480-68818dbfc559 is finished
2022-07-25 23:18:46,394 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b2b2b900-872c-4e46-965d-d9e8592a1f28, queue=onap-awx-0)
[2022-07-25 23:18:46,394: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b2b2b900-872c-4e46-965d-d9e8592a1f28, queue=onap-awx-0)
2022-07-25 23:18:46,481 DEBUG awx.main.dispatch delivered b2b2b900-872c-4e46-965d-d9e8592a1f28 to worker[180] qsize 0
2022-07-25 23:18:46,492 DEBUG awx.main.dispatch task b2b2b900-872c-4e46-965d-d9e8592a1f28 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:18:46,494 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:18:46,671 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(14ac85c6-4ad8-402f-bffd-d7c2c0784671, queue=onap-awx-0)
[2022-07-25 23:18:46,671: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(14ac85c6-4ad8-402f-bffd-d7c2c0784671, queue=onap-awx-0)
2022-07-25 23:18:46,702 DEBUG awx.main.dispatch delivered 14ac85c6-4ad8-402f-bffd-d7c2c0784671 to worker[181] qsize 0
2022-07-25 23:18:46,704 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3adc832f-23d5-49fd-b854-39f82d56b125, queue=awx_private_queue)
[2022-07-25 23:18:46,704: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3adc832f-23d5-49fd-b854-39f82d56b125, queue=awx_private_queue)
2022-07-25 23:18:46,707 DEBUG awx.main.dispatch task 14ac85c6-4ad8-402f-bffd-d7c2c0784671 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:18:46,724 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(12ab1f82-3af0-4ec7-9f99-73ce3aa673d4, queue=awx_private_queue)
[2022-07-25 23:18:46,724: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(12ab1f82-3af0-4ec7-9f99-73ce3aa673d4, queue=awx_private_queue)
2022-07-25 23:18:46,726 DEBUG awx.main.dispatch delivered 3adc832f-23d5-49fd-b854-39f82d56b125 to worker[183] qsize 0
2022-07-25 23:18:46,939 DEBUG awx.main.dispatch delivered 12ab1f82-3af0-4ec7-9f99-73ce3aa673d4 to worker[182] qsize 0
2022-07-25 23:18:46,939 DEBUG awx.main.dispatch task 3adc832f-23d5-49fd-b854-39f82d56b125 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:18:46,944 DEBUG awx.main.dispatch task 12ab1f82-3af0-4ec7-9f99-73ce3aa673d4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:18:46,947 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:18:46,980 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:18:46,989 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:18:16.756249+00:00
2022-07-25 23:18:47,013 DEBUG awx.main.dispatch task 14ac85c6-4ad8-402f-bffd-d7c2c0784671 is finished
2022-07-25 23:18:47,019 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:18:46,809 DEBUG awx.main.dispatch task b2b2b900-872c-4e46-965d-d9e8592a1f28 is finished
2022-07-25 23:18:47,261 DEBUG awx.main.dispatch task 12ab1f82-3af0-4ec7-9f99-73ce3aa673d4 is finished
2022-07-25 23:18:47,698 DEBUG awx.main.dispatch task 3adc832f-23d5-49fd-b854-39f82d56b125 is finished
RESULT 2
OKREADY
2022-07-25 23:19:06,727 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(91b24784-ba7f-46f2-9fc8-c414a84e4b38, queue=awx_private_queue)
[2022-07-25 23:19:06,727: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(91b24784-ba7f-46f2-9fc8-c414a84e4b38, queue=awx_private_queue)
2022-07-25 23:19:06,744 DEBUG awx.main.dispatch delivered 91b24784-ba7f-46f2-9fc8-c414a84e4b38 to worker[181] qsize 0
2022-07-25 23:19:06,746 DEBUG awx.main.dispatch task 91b24784-ba7f-46f2-9fc8-c414a84e4b38 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:19:06,748 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:19:06,855 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:19:06,962 DEBUG awx.main.dispatch task 91b24784-ba7f-46f2-9fc8-c414a84e4b38 is finished
2022-07-25 23:19:16,704 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(933e9e1f-0e54-4885-9ed8-01e5ee969e3a, queue=awx_private_queue)
[2022-07-25 23:19:16,704: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(933e9e1f-0e54-4885-9ed8-01e5ee969e3a, queue=awx_private_queue)
2022-07-25 23:19:16,721 DEBUG awx.main.dispatch delivered 933e9e1f-0e54-4885-9ed8-01e5ee969e3a to worker[183] qsize 0
2022-07-25 23:19:16,723 DEBUG awx.main.dispatch task 933e9e1f-0e54-4885-9ed8-01e5ee969e3a starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:19:16,846 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:19:16,916 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:18:46.980851+00:00
2022-07-25 23:19:17,696 DEBUG awx.main.dispatch task 933e9e1f-0e54-4885-9ed8-01e5ee969e3a is finished
2022-07-25 23:19:26,727 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0963e50d-c7d9-42cb-af4c-0e928703ec5b, queue=awx_private_queue)
[2022-07-25 23:19:26,727: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0963e50d-c7d9-42cb-af4c-0e928703ec5b, queue=awx_private_queue)
2022-07-25 23:19:26,761 DEBUG awx.main.dispatch delivered 0963e50d-c7d9-42cb-af4c-0e928703ec5b to worker[183] qsize 0
2022-07-25 23:19:26,764 DEBUG awx.main.dispatch task 0963e50d-c7d9-42cb-af4c-0e928703ec5b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:19:26,768 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:19:26,798 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:19:26,855 DEBUG awx.main.dispatch task 0963e50d-c7d9-42cb-af4c-0e928703ec5b is finished
2022-07-25 23:19:46,396 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6e76aee8-7cee-4ab8-b736-3718635494f7, queue=onap-awx-0)
[2022-07-25 23:19:46,396: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6e76aee8-7cee-4ab8-b736-3718635494f7, queue=onap-awx-0)
2022-07-25 23:19:46,530 DEBUG awx.main.dispatch delivered 6e76aee8-7cee-4ab8-b736-3718635494f7 to worker[181] qsize 0
2022-07-25 23:19:46,550 DEBUG awx.main.dispatch task 6e76aee8-7cee-4ab8-b736-3718635494f7 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:19:46,553 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:19:46,671 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(7912fe87-95fe-4ce2-aee0-799ba1cedca8, queue=onap-awx-0)
[2022-07-25 23:19:46,671: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(7912fe87-95fe-4ce2-aee0-799ba1cedca8, queue=onap-awx-0)
2022-07-25 23:19:46,691 DEBUG awx.main.dispatch delivered 7912fe87-95fe-4ce2-aee0-799ba1cedca8 to worker[183] qsize 0
2022-07-25 23:19:46,694 DEBUG awx.main.dispatch task 7912fe87-95fe-4ce2-aee0-799ba1cedca8 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:19:46,705 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(aac9c0c0-673c-4fb2-b673-6dae2cadf397, queue=awx_private_queue)
[2022-07-25 23:19:46,705: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(aac9c0c0-673c-4fb2-b673-6dae2cadf397, queue=awx_private_queue)
2022-07-25 23:19:46,721 DEBUG awx.main.dispatch delivered aac9c0c0-673c-4fb2-b673-6dae2cadf397 to worker[180] qsize 0
2022-07-25 23:19:46,724 DEBUG awx.main.dispatch task aac9c0c0-673c-4fb2-b673-6dae2cadf397 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:19:46,727 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(46e09225-f8d3-4d6e-9552-12a5c44f7c30, queue=awx_private_queue)
[2022-07-25 23:19:46,727: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(46e09225-f8d3-4d6e-9552-12a5c44f7c30, queue=awx_private_queue)
2022-07-25 23:19:46,817 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:19:46,827 DEBUG awx.main.dispatch task 7912fe87-95fe-4ce2-aee0-799ba1cedca8 is finished
2022-07-25 23:19:46,831 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:19:16.846694+00:00
2022-07-25 23:19:46,876 DEBUG awx.main.dispatch task 6e76aee8-7cee-4ab8-b736-3718635494f7 is finished
2022-07-25 23:19:46,745 DEBUG awx.main.dispatch delivered 46e09225-f8d3-4d6e-9552-12a5c44f7c30 to worker[182] qsize 0
2022-07-25 23:19:46,974 DEBUG awx.main.dispatch task 46e09225-f8d3-4d6e-9552-12a5c44f7c30 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:19:46,977 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:19:46,991 DEBUG awx.main.dispatch task aac9c0c0-673c-4fb2-b673-6dae2cadf397 is finished
2022-07-25 23:19:47,259 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:19:47,781 DEBUG awx.main.dispatch task 46e09225-f8d3-4d6e-9552-12a5c44f7c30 is finished
RESULT 2
OKREADY
2022-07-25 23:20:06,735 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(99cf9f48-439c-4251-ad06-aa955652e244, queue=awx_private_queue)
[2022-07-25 23:20:06,735: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(99cf9f48-439c-4251-ad06-aa955652e244, queue=awx_private_queue)
2022-07-25 23:20:06,768 DEBUG awx.main.dispatch delivered 99cf9f48-439c-4251-ad06-aa955652e244 to worker[182] qsize 0
2022-07-25 23:20:06,771 DEBUG awx.main.dispatch task 99cf9f48-439c-4251-ad06-aa955652e244 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:20:06,772 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:20:06,955 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:20:07,134 DEBUG awx.main.dispatch task 99cf9f48-439c-4251-ad06-aa955652e244 is finished
2022-07-25 23:20:16,704 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(94a49011-44e7-40c0-8d6b-9580f4b47ec6, queue=awx_private_queue)
[2022-07-25 23:20:16,704: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(94a49011-44e7-40c0-8d6b-9580f4b47ec6, queue=awx_private_queue)
2022-07-25 23:20:16,725 DEBUG awx.main.dispatch delivered 94a49011-44e7-40c0-8d6b-9580f4b47ec6 to worker[181] qsize 0
2022-07-25 23:20:16,727 DEBUG awx.main.dispatch task 94a49011-44e7-40c0-8d6b-9580f4b47ec6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:20:16,812 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:20:16,820 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:19:46.817685+00:00
2022-07-25 23:20:16,911 DEBUG awx.main.dispatch task 94a49011-44e7-40c0-8d6b-9580f4b47ec6 is finished
2022-07-25 23:20:26,735 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8aaa7fef-7892-47fd-9951-f79d1ccd6276, queue=awx_private_queue)
[2022-07-25 23:20:26,735: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8aaa7fef-7892-47fd-9951-f79d1ccd6276, queue=awx_private_queue)
2022-07-25 23:20:26,754 DEBUG awx.main.dispatch delivered 8aaa7fef-7892-47fd-9951-f79d1ccd6276 to worker[180] qsize 0
2022-07-25 23:20:26,758 DEBUG awx.main.dispatch task 8aaa7fef-7892-47fd-9951-f79d1ccd6276 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:20:26,760 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:20:26,861 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:20:27,117 DEBUG awx.main.dispatch task 8aaa7fef-7892-47fd-9951-f79d1ccd6276 is finished
2022-07-25 23:20:46,402 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b1e627e5-07c0-4dde-96a5-582c274eb5e4, queue=onap-awx-0)
[2022-07-25 23:20:46,402: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b1e627e5-07c0-4dde-96a5-582c274eb5e4, queue=onap-awx-0)
2022-07-25 23:20:46,516 DEBUG awx.main.dispatch delivered b1e627e5-07c0-4dde-96a5-582c274eb5e4 to worker[182] qsize 0
2022-07-25 23:20:46,518 DEBUG awx.main.dispatch task b1e627e5-07c0-4dde-96a5-582c274eb5e4 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:20:46,520 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:20:46,670 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(e248026f-d7ea-4b3e-b59c-7faca918bedb, queue=onap-awx-0)
[2022-07-25 23:20:46,670: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(e248026f-d7ea-4b3e-b59c-7faca918bedb, queue=onap-awx-0)
2022-07-25 23:20:46,783 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e5f6c6b5-e7fa-4e96-85e4-9e1934449124, queue=awx_private_queue)
[2022-07-25 23:20:46,783: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e5f6c6b5-e7fa-4e96-85e4-9e1934449124, queue=awx_private_queue)
2022-07-25 23:20:46,751 DEBUG awx.main.dispatch task b1e627e5-07c0-4dde-96a5-582c274eb5e4 is finished
2022-07-25 23:20:46,804 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f6133298-ea4a-4696-9a6c-3e5c019e356b, queue=awx_private_queue)
[2022-07-25 23:20:46,804: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f6133298-ea4a-4696-9a6c-3e5c019e356b, queue=awx_private_queue)
2022-07-25 23:20:46,779 DEBUG awx.main.dispatch delivered e248026f-d7ea-4b3e-b59c-7faca918bedb to worker[180] qsize 0
2022-07-25 23:20:46,994 DEBUG awx.main.dispatch task e248026f-d7ea-4b3e-b59c-7faca918bedb starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:20:46,995 DEBUG awx.main.dispatch delivered e5f6c6b5-e7fa-4e96-85e4-9e1934449124 to worker[181] qsize 0
2022-07-25 23:20:46,998 DEBUG awx.main.dispatch delivered f6133298-ea4a-4696-9a6c-3e5c019e356b to worker[183] qsize 0
2022-07-25 23:20:46,999 DEBUG awx.main.dispatch task e5f6c6b5-e7fa-4e96-85e4-9e1934449124 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:20:47,001 DEBUG awx.main.dispatch task f6133298-ea4a-4696-9a6c-3e5c019e356b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:20:47,004 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:20:47,128 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:20:47,131 DEBUG awx.main.dispatch task e248026f-d7ea-4b3e-b59c-7faca918bedb is finished
2022-07-25 23:20:47,138 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:20:16.812582+00:00
2022-07-25 23:20:47,174 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:20:47,247 DEBUG awx.main.dispatch task e5f6c6b5-e7fa-4e96-85e4-9e1934449124 is finished
2022-07-25 23:20:47,423 DEBUG awx.main.dispatch task f6133298-ea4a-4696-9a6c-3e5c019e356b is finished
RESULT 2
OKREADY
2022-07-25 23:21:06,813 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(96a82694-8b1e-4791-8889-ed98f4834c43, queue=awx_private_queue)
[2022-07-25 23:21:06,813: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(96a82694-8b1e-4791-8889-ed98f4834c43, queue=awx_private_queue)
2022-07-25 23:21:06,834 DEBUG awx.main.dispatch delivered 96a82694-8b1e-4791-8889-ed98f4834c43 to worker[180] qsize 0
2022-07-25 23:21:06,836 DEBUG awx.main.dispatch task 96a82694-8b1e-4791-8889-ed98f4834c43 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:21:06,837 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:21:06,875 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:21:06,935 DEBUG awx.main.dispatch task 96a82694-8b1e-4791-8889-ed98f4834c43 is finished
2022-07-25 23:21:16,784 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(821059d4-ad02-40d0-9f51-4a7d7f89f5b8, queue=awx_private_queue)
[2022-07-25 23:21:16,784: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(821059d4-ad02-40d0-9f51-4a7d7f89f5b8, queue=awx_private_queue)
2022-07-25 23:21:16,801 DEBUG awx.main.dispatch delivered 821059d4-ad02-40d0-9f51-4a7d7f89f5b8 to worker[181] qsize 0
2022-07-25 23:21:16,803 DEBUG awx.main.dispatch task 821059d4-ad02-40d0-9f51-4a7d7f89f5b8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:21:16,835 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:21:16,848 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:20:47.128794+00:00
2022-07-25 23:21:16,881 DEBUG awx.main.dispatch task 821059d4-ad02-40d0-9f51-4a7d7f89f5b8 is finished
2022-07-25 23:21:26,813 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5a48fbd1-8f94-4b8c-826b-7ae35084f2db, queue=awx_private_queue)
[2022-07-25 23:21:26,813: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5a48fbd1-8f94-4b8c-826b-7ae35084f2db, queue=awx_private_queue)
2022-07-25 23:21:26,831 DEBUG awx.main.dispatch delivered 5a48fbd1-8f94-4b8c-826b-7ae35084f2db to worker[180] qsize 0
2022-07-25 23:21:26,834 DEBUG awx.main.dispatch task 5a48fbd1-8f94-4b8c-826b-7ae35084f2db starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:21:26,836 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:21:26,904 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:21:26,977 DEBUG awx.main.dispatch task 5a48fbd1-8f94-4b8c-826b-7ae35084f2db is finished
2022-07-25 23:21:46,404 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6afe1177-03ad-44c5-ba52-f2c4ab45fb89, queue=onap-awx-0)
[2022-07-25 23:21:46,404: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6afe1177-03ad-44c5-ba52-f2c4ab45fb89, queue=onap-awx-0)
2022-07-25 23:21:46,446 DEBUG awx.main.dispatch delivered 6afe1177-03ad-44c5-ba52-f2c4ab45fb89 to worker[180] qsize 0
2022-07-25 23:21:46,449 DEBUG awx.main.dispatch task 6afe1177-03ad-44c5-ba52-f2c4ab45fb89 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:21:46,450 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:21:46,494 DEBUG awx.main.dispatch task 6afe1177-03ad-44c5-ba52-f2c4ab45fb89 is finished
2022-07-25 23:21:46,670 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(86156a8a-19fa-4a48-95b7-ad3dc00d2a48, queue=onap-awx-0)
[2022-07-25 23:21:46,670: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(86156a8a-19fa-4a48-95b7-ad3dc00d2a48, queue=onap-awx-0)
2022-07-25 23:21:46,712 DEBUG awx.main.dispatch delivered 86156a8a-19fa-4a48-95b7-ad3dc00d2a48 to worker[183] qsize 0
2022-07-25 23:21:46,714 DEBUG awx.main.dispatch task 86156a8a-19fa-4a48-95b7-ad3dc00d2a48 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:21:46,736 DEBUG awx.main.dispatch task 86156a8a-19fa-4a48-95b7-ad3dc00d2a48 is finished
2022-07-25 23:21:46,784 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(485a2ecb-1d00-4f43-8486-11d8a608d4f5, queue=awx_private_queue)
[2022-07-25 23:21:46,784: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(485a2ecb-1d00-4f43-8486-11d8a608d4f5, queue=awx_private_queue)
2022-07-25 23:21:46,813 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4dd553ea-d926-45f2-84f1-13217fa51182, queue=awx_private_queue)
[2022-07-25 23:21:46,813: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4dd553ea-d926-45f2-84f1-13217fa51182, queue=awx_private_queue)
2022-07-25 23:21:46,802 DEBUG awx.main.dispatch delivered 485a2ecb-1d00-4f43-8486-11d8a608d4f5 to worker[182] qsize 0
2022-07-25 23:21:47,007 DEBUG awx.main.dispatch delivered 4dd553ea-d926-45f2-84f1-13217fa51182 to worker[183] qsize 0
2022-07-25 23:21:47,007 DEBUG awx.main.dispatch task 485a2ecb-1d00-4f43-8486-11d8a608d4f5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:21:47,009 DEBUG awx.main.dispatch task 4dd553ea-d926-45f2-84f1-13217fa51182 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:21:47,011 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:21:47,029 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:21:47,034 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:21:16.835908+00:00
2022-07-25 23:21:47,052 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:21:47,056 DEBUG awx.main.dispatch task 485a2ecb-1d00-4f43-8486-11d8a608d4f5 is finished
2022-07-25 23:21:47,113 DEBUG awx.main.dispatch task 4dd553ea-d926-45f2-84f1-13217fa51182 is finished
RESULT 2
OKREADY
2022-07-25 23:22:06,824 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cb4176ca-4e0f-491a-947a-2c3d0036b70d, queue=awx_private_queue)
[2022-07-25 23:22:06,824: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cb4176ca-4e0f-491a-947a-2c3d0036b70d, queue=awx_private_queue)
2022-07-25 23:22:06,859 DEBUG awx.main.dispatch delivered cb4176ca-4e0f-491a-947a-2c3d0036b70d to worker[183] qsize 0
2022-07-25 23:22:06,865 DEBUG awx.main.dispatch task cb4176ca-4e0f-491a-947a-2c3d0036b70d starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:22:06,868 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:22:06,930 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:22:07,000 DEBUG awx.main.dispatch task cb4176ca-4e0f-491a-947a-2c3d0036b70d is finished
2022-07-25 23:22:16,784 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ddd0aee9-8219-45ea-932e-8990b693145b, queue=awx_private_queue)
[2022-07-25 23:22:16,784: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ddd0aee9-8219-45ea-932e-8990b693145b, queue=awx_private_queue)
2022-07-25 23:22:16,817 DEBUG awx.main.dispatch delivered ddd0aee9-8219-45ea-932e-8990b693145b to worker[183] qsize 0
2022-07-25 23:22:16,820 DEBUG awx.main.dispatch task ddd0aee9-8219-45ea-932e-8990b693145b starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:22:16,842 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:22:16,847 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:21:47.030052+00:00
2022-07-25 23:22:16,869 DEBUG awx.main.dispatch task ddd0aee9-8219-45ea-932e-8990b693145b is finished
2022-07-25 23:22:26,824 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(759044fa-30b5-4111-9480-b04a9cab7d09, queue=awx_private_queue)
[2022-07-25 23:22:26,824: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(759044fa-30b5-4111-9480-b04a9cab7d09, queue=awx_private_queue)
2022-07-25 23:22:26,860 DEBUG awx.main.dispatch delivered 759044fa-30b5-4111-9480-b04a9cab7d09 to worker[183] qsize 0
2022-07-25 23:22:26,862 DEBUG awx.main.dispatch task 759044fa-30b5-4111-9480-b04a9cab7d09 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:22:26,864 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:22:26,915 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:22:26,994 DEBUG awx.main.dispatch task 759044fa-30b5-4111-9480-b04a9cab7d09 is finished
2022-07-25 23:22:46,415 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(17d207a7-681e-4bfa-a91f-d72336a7cf20, queue=onap-awx-0)
[2022-07-25 23:22:46,415: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(17d207a7-681e-4bfa-a91f-d72336a7cf20, queue=onap-awx-0)
2022-07-25 23:22:46,456 DEBUG awx.main.dispatch delivered 17d207a7-681e-4bfa-a91f-d72336a7cf20 to worker[183] qsize 0
2022-07-25 23:22:46,460 DEBUG awx.main.dispatch task 17d207a7-681e-4bfa-a91f-d72336a7cf20 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:22:46,463 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:22:46,508 DEBUG awx.main.dispatch task 17d207a7-681e-4bfa-a91f-d72336a7cf20 is finished
2022-07-25 23:22:46,670 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(16be249d-97c9-4314-8c11-265f564fe1a5, queue=onap-awx-0)
[2022-07-25 23:22:46,670: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(16be249d-97c9-4314-8c11-265f564fe1a5, queue=onap-awx-0)
2022-07-25 23:22:46,687 DEBUG awx.main.dispatch delivered 16be249d-97c9-4314-8c11-265f564fe1a5 to worker[183] qsize 0
2022-07-25 23:22:46,695 DEBUG awx.main.dispatch task 16be249d-97c9-4314-8c11-265f564fe1a5 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:22:46,737 DEBUG awx.main.dispatch task 16be249d-97c9-4314-8c11-265f564fe1a5 is finished
2022-07-25 23:22:46,784 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3645a175-5b68-4f3e-a18b-4b91394188b6, queue=awx_private_queue)
[2022-07-25 23:22:46,784: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3645a175-5b68-4f3e-a18b-4b91394188b6, queue=awx_private_queue)
2022-07-25 23:22:46,824 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6671dd55-5633-4fde-ba31-86219b68176a, queue=awx_private_queue)
[2022-07-25 23:22:46,824: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6671dd55-5633-4fde-ba31-86219b68176a, queue=awx_private_queue)
2022-07-25 23:22:46,804 DEBUG awx.main.dispatch delivered 3645a175-5b68-4f3e-a18b-4b91394188b6 to worker[182] qsize 0
2022-07-25 23:22:47,034 DEBUG awx.main.dispatch delivered 6671dd55-5633-4fde-ba31-86219b68176a to worker[181] qsize 0
2022-07-25 23:22:47,035 DEBUG awx.main.dispatch task 3645a175-5b68-4f3e-a18b-4b91394188b6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:22:47,037 DEBUG awx.main.dispatch task 6671dd55-5633-4fde-ba31-86219b68176a starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:22:47,042 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:22:47,107 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:22:47,120 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:22:16.842923+00:00
2022-07-25 23:22:47,147 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:22:47,156 DEBUG awx.main.dispatch task 3645a175-5b68-4f3e-a18b-4b91394188b6 is finished
2022-07-25 23:22:47,202 DEBUG awx.main.dispatch task 6671dd55-5633-4fde-ba31-86219b68176a is finished
RESULT 2
OKREADY
2022-07-25 23:23:06,824 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(19de7050-e776-4c60-87cc-f3d071cc3f82, queue=awx_private_queue)
[2022-07-25 23:23:06,824: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(19de7050-e776-4c60-87cc-f3d071cc3f82, queue=awx_private_queue)
2022-07-25 23:23:06,841 DEBUG awx.main.dispatch delivered 19de7050-e776-4c60-87cc-f3d071cc3f82 to worker[183] qsize 0
2022-07-25 23:23:06,844 DEBUG awx.main.dispatch task 19de7050-e776-4c60-87cc-f3d071cc3f82 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:23:06,846 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:23:06,874 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:23:06,958 DEBUG awx.main.dispatch task 19de7050-e776-4c60-87cc-f3d071cc3f82 is finished
2022-07-25 23:23:16,784 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a2c80611-892f-4cf2-bd2f-5173861e061a, queue=awx_private_queue)
[2022-07-25 23:23:16,784: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a2c80611-892f-4cf2-bd2f-5173861e061a, queue=awx_private_queue)
2022-07-25 23:23:16,800 DEBUG awx.main.dispatch delivered a2c80611-892f-4cf2-bd2f-5173861e061a to worker[183] qsize 0
2022-07-25 23:23:16,803 DEBUG awx.main.dispatch task a2c80611-892f-4cf2-bd2f-5173861e061a starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:23:16,824 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:23:16,828 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:22:47.107642+00:00
2022-07-25 23:23:16,852 DEBUG awx.main.dispatch task a2c80611-892f-4cf2-bd2f-5173861e061a is finished
2022-07-25 23:23:26,825 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a6ea1b65-c712-409e-a5ce-59084cfb2efc, queue=awx_private_queue)
[2022-07-25 23:23:26,825: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a6ea1b65-c712-409e-a5ce-59084cfb2efc, queue=awx_private_queue)
2022-07-25 23:23:26,846 DEBUG awx.main.dispatch delivered a6ea1b65-c712-409e-a5ce-59084cfb2efc to worker[180] qsize 0
2022-07-25 23:23:26,849 DEBUG awx.main.dispatch task a6ea1b65-c712-409e-a5ce-59084cfb2efc starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:23:26,853 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:23:26,913 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:23:27,012 DEBUG awx.main.dispatch task a6ea1b65-c712-409e-a5ce-59084cfb2efc is finished
2022-07-25 23:23:46,423 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(2a0314d5-72dc-4f15-9148-319d7cc0a60d, queue=onap-awx-0)
[2022-07-25 23:23:46,423: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(2a0314d5-72dc-4f15-9148-319d7cc0a60d, queue=onap-awx-0)
2022-07-25 23:23:46,464 DEBUG awx.main.dispatch delivered 2a0314d5-72dc-4f15-9148-319d7cc0a60d to worker[182] qsize 0
2022-07-25 23:23:46,467 DEBUG awx.main.dispatch task 2a0314d5-72dc-4f15-9148-319d7cc0a60d starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:23:46,469 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:23:46,520 DEBUG awx.main.dispatch task 2a0314d5-72dc-4f15-9148-319d7cc0a60d is finished
2022-07-25 23:23:46,671 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(057df8a9-54a2-43c8-a2ba-0a315a7e794b, queue=onap-awx-0)
[2022-07-25 23:23:46,671: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(057df8a9-54a2-43c8-a2ba-0a315a7e794b, queue=onap-awx-0)
2022-07-25 23:23:46,687 DEBUG awx.main.dispatch delivered 057df8a9-54a2-43c8-a2ba-0a315a7e794b to worker[180] qsize 0
2022-07-25 23:23:46,690 DEBUG awx.main.dispatch task 057df8a9-54a2-43c8-a2ba-0a315a7e794b starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:23:46,713 DEBUG awx.main.dispatch task 057df8a9-54a2-43c8-a2ba-0a315a7e794b is finished
2022-07-25 23:23:46,784 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4029e43f-4848-4b78-95d9-0fe58a124b59, queue=awx_private_queue)
[2022-07-25 23:23:46,784: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4029e43f-4848-4b78-95d9-0fe58a124b59, queue=awx_private_queue)
2022-07-25 23:23:46,832 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d6b3b82a-0f85-4684-9d1c-2151a5bf33af, queue=awx_private_queue)
[2022-07-25 23:23:46,832: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d6b3b82a-0f85-4684-9d1c-2151a5bf33af, queue=awx_private_queue)
2022-07-25 23:23:46,830 DEBUG awx.main.dispatch delivered 4029e43f-4848-4b78-95d9-0fe58a124b59 to worker[180] qsize 0
2022-07-25 23:23:47,045 DEBUG awx.main.dispatch task 4029e43f-4848-4b78-95d9-0fe58a124b59 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:23:47,046 DEBUG awx.main.dispatch delivered d6b3b82a-0f85-4684-9d1c-2151a5bf33af to worker[182] qsize 0
2022-07-25 23:23:47,049 DEBUG awx.main.dispatch task d6b3b82a-0f85-4684-9d1c-2151a5bf33af starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:23:47,052 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:23:47,064 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:23:47,069 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:23:16.824748+00:00
2022-07-25 23:23:47,094 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:23:47,114 DEBUG awx.main.dispatch task 4029e43f-4848-4b78-95d9-0fe58a124b59 is finished
2022-07-25 23:23:47,177 DEBUG awx.main.dispatch task d6b3b82a-0f85-4684-9d1c-2151a5bf33af is finished
RESULT 2
OKREADY
2022-07-25 23:24:06,832 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0435e1f7-d223-45aa-bb30-15962adb1dc9, queue=awx_private_queue)
[2022-07-25 23:24:06,832: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0435e1f7-d223-45aa-bb30-15962adb1dc9, queue=awx_private_queue)
2022-07-25 23:24:06,851 DEBUG awx.main.dispatch delivered 0435e1f7-d223-45aa-bb30-15962adb1dc9 to worker[180] qsize 0
2022-07-25 23:24:06,854 DEBUG awx.main.dispatch task 0435e1f7-d223-45aa-bb30-15962adb1dc9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:24:06,856 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:24:06,922 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:24:07,001 DEBUG awx.main.dispatch task 0435e1f7-d223-45aa-bb30-15962adb1dc9 is finished
2022-07-25 23:24:16,784 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2a4d1e18-7688-4edb-9ade-ff10723b3182, queue=awx_private_queue)
[2022-07-25 23:24:16,784: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2a4d1e18-7688-4edb-9ade-ff10723b3182, queue=awx_private_queue)
2022-07-25 23:24:16,805 DEBUG awx.main.dispatch delivered 2a4d1e18-7688-4edb-9ade-ff10723b3182 to worker[181] qsize 0
2022-07-25 23:24:16,808 DEBUG awx.main.dispatch task 2a4d1e18-7688-4edb-9ade-ff10723b3182 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:24:16,838 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:24:16,842 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:23:47.064822+00:00
2022-07-25 23:24:16,863 DEBUG awx.main.dispatch task 2a4d1e18-7688-4edb-9ade-ff10723b3182 is finished
2022-07-25 23:24:26,832 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(975a9290-1f39-446a-8876-f0a902e5ffed, queue=awx_private_queue)
[2022-07-25 23:24:26,832: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(975a9290-1f39-446a-8876-f0a902e5ffed, queue=awx_private_queue)
2022-07-25 23:24:26,864 DEBUG awx.main.dispatch delivered 975a9290-1f39-446a-8876-f0a902e5ffed to worker[183] qsize 0
2022-07-25 23:24:26,867 DEBUG awx.main.dispatch task 975a9290-1f39-446a-8876-f0a902e5ffed starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:24:26,869 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:24:26,935 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:24:27,131 DEBUG awx.main.dispatch task 975a9290-1f39-446a-8876-f0a902e5ffed is finished
2022-07-25 23:24:46,434 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(25510066-d4c5-4549-aaab-b50bef8545df, queue=onap-awx-0)
[2022-07-25 23:24:46,434: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(25510066-d4c5-4549-aaab-b50bef8545df, queue=onap-awx-0)
2022-07-25 23:24:46,480 DEBUG awx.main.dispatch delivered 25510066-d4c5-4549-aaab-b50bef8545df to worker[181] qsize 0
2022-07-25 23:24:46,484 DEBUG awx.main.dispatch task 25510066-d4c5-4549-aaab-b50bef8545df starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:24:46,486 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:24:46,532 DEBUG awx.main.dispatch task 25510066-d4c5-4549-aaab-b50bef8545df is finished
2022-07-25 23:24:46,670 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(cf808d67-ce88-4166-837e-939877a3aea4, queue=onap-awx-0)
[2022-07-25 23:24:46,670: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(cf808d67-ce88-4166-837e-939877a3aea4, queue=onap-awx-0)
2022-07-25 23:24:46,688 DEBUG awx.main.dispatch delivered cf808d67-ce88-4166-837e-939877a3aea4 to worker[183] qsize 0
2022-07-25 23:24:46,691 DEBUG awx.main.dispatch task cf808d67-ce88-4166-837e-939877a3aea4 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:24:46,717 DEBUG awx.main.dispatch task cf808d67-ce88-4166-837e-939877a3aea4 is finished
2022-07-25 23:24:46,784 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8693f0b8-c4d2-46cf-bbd3-197a229065ab, queue=awx_private_queue)
[2022-07-25 23:24:46,784: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8693f0b8-c4d2-46cf-bbd3-197a229065ab, queue=awx_private_queue)
2022-07-25 23:24:46,804 DEBUG awx.main.dispatch delivered 8693f0b8-c4d2-46cf-bbd3-197a229065ab to worker[182] qsize 0
2022-07-25 23:24:46,823 DEBUG awx.main.dispatch task 8693f0b8-c4d2-46cf-bbd3-197a229065ab starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:24:46,833 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1c4142f6-f01c-4f71-971e-5dcab1313aa0, queue=awx_private_queue)
[2022-07-25 23:24:46,833: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1c4142f6-f01c-4f71-971e-5dcab1313aa0, queue=awx_private_queue)
2022-07-25 23:24:46,866 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:24:46,870 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:24:16.838588+00:00
2022-07-25 23:24:46,916 DEBUG awx.main.dispatch task 8693f0b8-c4d2-46cf-bbd3-197a229065ab is finished
2022-07-25 23:24:46,850 DEBUG awx.main.dispatch delivered 1c4142f6-f01c-4f71-971e-5dcab1313aa0 to worker[180] qsize 0
2022-07-25 23:24:47,061 DEBUG awx.main.dispatch task 1c4142f6-f01c-4f71-971e-5dcab1313aa0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:24:47,064 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:24:47,117 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:24:47,172 DEBUG awx.main.dispatch task 1c4142f6-f01c-4f71-971e-5dcab1313aa0 is finished
RESULT 2
OKREADY
2022-07-25 23:25:06,842 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1d567b18-0353-4317-921c-0d5374fd1a40, queue=awx_private_queue)
[2022-07-25 23:25:06,842: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1d567b18-0353-4317-921c-0d5374fd1a40, queue=awx_private_queue)
2022-07-25 23:25:06,862 DEBUG awx.main.dispatch delivered 1d567b18-0353-4317-921c-0d5374fd1a40 to worker[181] qsize 0
2022-07-25 23:25:06,866 DEBUG awx.main.dispatch task 1d567b18-0353-4317-921c-0d5374fd1a40 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:25:06,871 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:25:06,905 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:25:06,962 DEBUG awx.main.dispatch task 1d567b18-0353-4317-921c-0d5374fd1a40 is finished
2022-07-25 23:25:16,784 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(686e7e52-f613-4867-988d-75efabe25d71, queue=awx_private_queue)
[2022-07-25 23:25:16,784: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(686e7e52-f613-4867-988d-75efabe25d71, queue=awx_private_queue)
2022-07-25 23:25:16,816 DEBUG awx.main.dispatch delivered 686e7e52-f613-4867-988d-75efabe25d71 to worker[180] qsize 0
2022-07-25 23:25:16,818 DEBUG awx.main.dispatch task 686e7e52-f613-4867-988d-75efabe25d71 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:25:16,862 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:25:16,867 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:24:46.866551+00:00
2022-07-25 23:25:16,902 DEBUG awx.main.dispatch task 686e7e52-f613-4867-988d-75efabe25d71 is finished
2022-07-25 23:25:26,840 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0e72a261-e851-4763-9a29-f487d9d9b267, queue=awx_private_queue)
[2022-07-25 23:25:26,840: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0e72a261-e851-4763-9a29-f487d9d9b267, queue=awx_private_queue)
2022-07-25 23:25:26,858 DEBUG awx.main.dispatch delivered 0e72a261-e851-4763-9a29-f487d9d9b267 to worker[180] qsize 0
2022-07-25 23:25:26,860 DEBUG awx.main.dispatch task 0e72a261-e851-4763-9a29-f487d9d9b267 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:25:26,862 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:25:26,925 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:25:26,987 DEBUG awx.main.dispatch task 0e72a261-e851-4763-9a29-f487d9d9b267 is finished
2022-07-25 23:25:46,443 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c86de172-cce5-4e31-a1df-3ae17ce4cc42, queue=onap-awx-0)
[2022-07-25 23:25:46,443: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c86de172-cce5-4e31-a1df-3ae17ce4cc42, queue=onap-awx-0)
2022-07-25 23:25:46,485 DEBUG awx.main.dispatch delivered c86de172-cce5-4e31-a1df-3ae17ce4cc42 to worker[180] qsize 0
2022-07-25 23:25:46,489 DEBUG awx.main.dispatch task c86de172-cce5-4e31-a1df-3ae17ce4cc42 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:25:46,490 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:25:46,530 DEBUG awx.main.dispatch task c86de172-cce5-4e31-a1df-3ae17ce4cc42 is finished
2022-07-25 23:25:46,670 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(db0fa9c6-ef1c-406f-9430-32b25309cf54, queue=onap-awx-0)
[2022-07-25 23:25:46,670: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(db0fa9c6-ef1c-406f-9430-32b25309cf54, queue=onap-awx-0)
2022-07-25 23:25:46,688 DEBUG awx.main.dispatch delivered db0fa9c6-ef1c-406f-9430-32b25309cf54 to worker[181] qsize 0
2022-07-25 23:25:46,690 DEBUG awx.main.dispatch task db0fa9c6-ef1c-406f-9430-32b25309cf54 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:25:46,721 DEBUG awx.main.dispatch task db0fa9c6-ef1c-406f-9430-32b25309cf54 is finished
2022-07-25 23:25:46,785 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7b79fedc-9eb9-4397-864c-aa26d5e03102, queue=awx_private_queue)
[2022-07-25 23:25:46,785: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7b79fedc-9eb9-4397-864c-aa26d5e03102, queue=awx_private_queue)
2022-07-25 23:25:46,813 DEBUG awx.main.dispatch delivered 7b79fedc-9eb9-4397-864c-aa26d5e03102 to worker[182] qsize 0
2022-07-25 23:25:46,817 DEBUG awx.main.dispatch task 7b79fedc-9eb9-4397-864c-aa26d5e03102 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:25:46,841 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2a6150ec-8669-4176-880f-1c69a91a3d8c, queue=awx_private_queue)
[2022-07-25 23:25:46,841: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2a6150ec-8669-4176-880f-1c69a91a3d8c, queue=awx_private_queue)
2022-07-25 23:25:46,844 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:25:46,848 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:25:16.863111+00:00
2022-07-25 23:25:46,876 DEBUG awx.main.dispatch task 7b79fedc-9eb9-4397-864c-aa26d5e03102 is finished
2022-07-25 23:25:46,860 DEBUG awx.main.dispatch delivered 2a6150ec-8669-4176-880f-1c69a91a3d8c to worker[183] qsize 0
2022-07-25 23:25:47,078 DEBUG awx.main.dispatch task 2a6150ec-8669-4176-880f-1c69a91a3d8c starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:25:47,081 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:25:47,121 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:25:47,182 DEBUG awx.main.dispatch task 2a6150ec-8669-4176-880f-1c69a91a3d8c is finished
RESULT 2
OKREADY
2022-07-25 23:26:06,852 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1fb54438-d2a2-47ad-b699-38d040ed6eb6, queue=awx_private_queue)
[2022-07-25 23:26:06,852: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1fb54438-d2a2-47ad-b699-38d040ed6eb6, queue=awx_private_queue)
2022-07-25 23:26:06,870 DEBUG awx.main.dispatch delivered 1fb54438-d2a2-47ad-b699-38d040ed6eb6 to worker[183] qsize 0
2022-07-25 23:26:06,872 DEBUG awx.main.dispatch task 1fb54438-d2a2-47ad-b699-38d040ed6eb6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:26:06,874 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:26:06,912 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:26:06,976 DEBUG awx.main.dispatch task 1fb54438-d2a2-47ad-b699-38d040ed6eb6 is finished
2022-07-25 23:26:16,785 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f67b384b-ed1a-41f6-9da6-5e5d2a809f52, queue=awx_private_queue)
[2022-07-25 23:26:16,785: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f67b384b-ed1a-41f6-9da6-5e5d2a809f52, queue=awx_private_queue)
2022-07-25 23:26:16,820 DEBUG awx.main.dispatch delivered f67b384b-ed1a-41f6-9da6-5e5d2a809f52 to worker[180] qsize 0
2022-07-25 23:26:16,823 DEBUG awx.main.dispatch task f67b384b-ed1a-41f6-9da6-5e5d2a809f52 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:26:16,842 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:26:16,847 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:25:46.844710+00:00
2022-07-25 23:26:16,866 DEBUG awx.main.dispatch task f67b384b-ed1a-41f6-9da6-5e5d2a809f52 is finished
2022-07-25 23:26:26,852 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5aef7881-b111-4069-8b05-44260d57212f, queue=awx_private_queue)
[2022-07-25 23:26:26,852: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5aef7881-b111-4069-8b05-44260d57212f, queue=awx_private_queue)
2022-07-25 23:26:26,875 DEBUG awx.main.dispatch delivered 5aef7881-b111-4069-8b05-44260d57212f to worker[182] qsize 0
2022-07-25 23:26:26,877 DEBUG awx.main.dispatch task 5aef7881-b111-4069-8b05-44260d57212f starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:26:26,879 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:26:26,907 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:26:26,965 DEBUG awx.main.dispatch task 5aef7881-b111-4069-8b05-44260d57212f is finished
2022-07-25 23:26:46,391 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(195b9aab-c16b-462c-9adf-159e67cecbd3, queue=onap-awx-0)
[2022-07-25 23:26:46,391: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(195b9aab-c16b-462c-9adf-159e67cecbd3, queue=onap-awx-0)
2022-07-25 23:26:46,455 DEBUG awx.main.dispatch delivered 195b9aab-c16b-462c-9adf-159e67cecbd3 to worker[181] qsize 0
2022-07-25 23:26:46,460 DEBUG awx.main.dispatch task 195b9aab-c16b-462c-9adf-159e67cecbd3 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2022-07-25 23:26:46,462 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2022-07-25 23:26:46,498 DEBUG awx.main.dispatch task 195b9aab-c16b-462c-9adf-159e67cecbd3 is finished
2022-07-25 23:26:46,456 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(97a722c2-2865-4378-a822-7a802265ae4e, queue=onap-awx-0)
[2022-07-25 23:26:46,456: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(97a722c2-2865-4378-a822-7a802265ae4e, queue=onap-awx-0)
2022-07-25 23:26:46,704 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3193140c-e3e0-48b3-ba4c-273c34d20795, queue=onap-awx-0)
[2022-07-25 23:26:46,704: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3193140c-e3e0-48b3-ba4c-273c34d20795, queue=onap-awx-0)
2022-07-25 23:26:46,735 DEBUG awx.main.dispatch delivered 97a722c2-2865-4378-a822-7a802265ae4e to worker[183] qsize 0
2022-07-25 23:26:46,738 DEBUG awx.main.dispatch delivered 3193140c-e3e0-48b3-ba4c-273c34d20795 to worker[180] qsize 0
2022-07-25 23:26:46,738 DEBUG awx.main.dispatch task 97a722c2-2865-4378-a822-7a802265ae4e starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:26:46,740 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:26:46,740 DEBUG awx.main.dispatch task 3193140c-e3e0-48b3-ba4c-273c34d20795 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:26:46,763 DEBUG awx.main.dispatch task 3193140c-e3e0-48b3-ba4c-273c34d20795 is finished
2022-07-25 23:26:46,780 DEBUG awx.main.dispatch task 97a722c2-2865-4378-a822-7a802265ae4e is finished
2022-07-25 23:26:46,784 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(20a51f79-8ef9-448a-b7c7-72d2535292f3, queue=awx_private_queue)
[2022-07-25 23:26:46,784: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(20a51f79-8ef9-448a-b7c7-72d2535292f3, queue=awx_private_queue)
2022-07-25 23:26:46,805 DEBUG awx.main.dispatch delivered 20a51f79-8ef9-448a-b7c7-72d2535292f3 to worker[183] qsize 0
2022-07-25 23:26:46,808 DEBUG awx.main.dispatch task 20a51f79-8ef9-448a-b7c7-72d2535292f3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:26:46,825 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:26:46,830 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:26:16.843157+00:00
2022-07-25 23:26:46,852 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c1137f87-7278-4478-94ac-79cbd554599e, queue=awx_private_queue)
[2022-07-25 23:26:46,852: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c1137f87-7278-4478-94ac-79cbd554599e, queue=awx_private_queue)
2022-07-25 23:26:46,854 DEBUG awx.main.dispatch task 20a51f79-8ef9-448a-b7c7-72d2535292f3 is finished
2022-07-25 23:26:46,871 DEBUG awx.main.dispatch delivered c1137f87-7278-4478-94ac-79cbd554599e to worker[182] qsize 0
2022-07-25 23:26:46,876 DEBUG awx.main.dispatch task c1137f87-7278-4478-94ac-79cbd554599e starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:26:46,879 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:26:46,913 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:26:46,965 DEBUG awx.main.dispatch task c1137f87-7278-4478-94ac-79cbd554599e is finished
RESULT 2
OKREADY
2022-07-25 23:27:06,860 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f38ca54c-447e-45de-b146-c5d1dcf1a2fa, queue=awx_private_queue)
[2022-07-25 23:27:06,860: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f38ca54c-447e-45de-b146-c5d1dcf1a2fa, queue=awx_private_queue)
2022-07-25 23:27:06,882 DEBUG awx.main.dispatch delivered f38ca54c-447e-45de-b146-c5d1dcf1a2fa to worker[182] qsize 0
2022-07-25 23:27:06,887 DEBUG awx.main.dispatch task f38ca54c-447e-45de-b146-c5d1dcf1a2fa starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:27:06,889 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:27:06,943 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:27:07,007 DEBUG awx.main.dispatch task f38ca54c-447e-45de-b146-c5d1dcf1a2fa is finished
2022-07-25 23:27:16,786 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e64307fe-19cd-45ce-b795-45bf748adb84, queue=awx_private_queue)
[2022-07-25 23:27:16,786: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e64307fe-19cd-45ce-b795-45bf748adb84, queue=awx_private_queue)
2022-07-25 23:27:16,825 DEBUG awx.main.dispatch delivered e64307fe-19cd-45ce-b795-45bf748adb84 to worker[183] qsize 0
2022-07-25 23:27:16,829 DEBUG awx.main.dispatch task e64307fe-19cd-45ce-b795-45bf748adb84 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:27:16,849 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:27:16,855 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:26:46.825814+00:00
2022-07-25 23:27:16,880 DEBUG awx.main.dispatch task e64307fe-19cd-45ce-b795-45bf748adb84 is finished
2022-07-25 23:27:26,859 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(218c6721-fc91-4b0a-85e9-55d80fdc371d, queue=awx_private_queue)
[2022-07-25 23:27:26,859: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(218c6721-fc91-4b0a-85e9-55d80fdc371d, queue=awx_private_queue)
2022-07-25 23:27:26,879 DEBUG awx.main.dispatch delivered 218c6721-fc91-4b0a-85e9-55d80fdc371d to worker[180] qsize 0
2022-07-25 23:27:26,882 DEBUG awx.main.dispatch task 218c6721-fc91-4b0a-85e9-55d80fdc371d starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:27:26,885 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:27:26,931 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:27:27,003 DEBUG awx.main.dispatch task 218c6721-fc91-4b0a-85e9-55d80fdc371d is finished
2022-07-25 23:27:46,465 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(20f80083-816d-42a4-a961-a7185cd82462, queue=onap-awx-0)
[2022-07-25 23:27:46,465: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(20f80083-816d-42a4-a961-a7185cd82462, queue=onap-awx-0)
2022-07-25 23:27:46,731 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(99a8b1ed-383b-41dd-90bb-4a4b9ea2177c, queue=onap-awx-0)
[2022-07-25 23:27:46,731: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(99a8b1ed-383b-41dd-90bb-4a4b9ea2177c, queue=onap-awx-0)
2022-07-25 23:27:46,751 DEBUG awx.main.dispatch delivered 20f80083-816d-42a4-a961-a7185cd82462 to worker[182] qsize 0
2022-07-25 23:27:46,753 DEBUG awx.main.dispatch delivered 99a8b1ed-383b-41dd-90bb-4a4b9ea2177c to worker[181] qsize 0
2022-07-25 23:27:46,753 DEBUG awx.main.dispatch task 20f80083-816d-42a4-a961-a7185cd82462 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:27:46,755 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:27:46,756 DEBUG awx.main.dispatch task 99a8b1ed-383b-41dd-90bb-4a4b9ea2177c starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:27:46,785 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(18cefbe9-fb38-44ae-8d5c-bfd1ab4bfbe9, queue=awx_private_queue)
[2022-07-25 23:27:46,785: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(18cefbe9-fb38-44ae-8d5c-bfd1ab4bfbe9, queue=awx_private_queue)
2022-07-25 23:27:46,795 DEBUG awx.main.dispatch task 99a8b1ed-383b-41dd-90bb-4a4b9ea2177c is finished
2022-07-25 23:27:46,803 DEBUG awx.main.dispatch task 20f80083-816d-42a4-a961-a7185cd82462 is finished
2022-07-25 23:27:46,806 DEBUG awx.main.dispatch delivered 18cefbe9-fb38-44ae-8d5c-bfd1ab4bfbe9 to worker[181] qsize 0
2022-07-25 23:27:46,809 DEBUG awx.main.dispatch task 18cefbe9-fb38-44ae-8d5c-bfd1ab4bfbe9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:27:46,825 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:27:46,830 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:27:16.849442+00:00
2022-07-25 23:27:46,853 DEBUG awx.main.dispatch task 18cefbe9-fb38-44ae-8d5c-bfd1ab4bfbe9 is finished
2022-07-25 23:27:46,860 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(53f0886d-1db8-4308-8c17-a75ca4479f30, queue=awx_private_queue)
[2022-07-25 23:27:46,860: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(53f0886d-1db8-4308-8c17-a75ca4479f30, queue=awx_private_queue)
2022-07-25 23:27:46,880 DEBUG awx.main.dispatch delivered 53f0886d-1db8-4308-8c17-a75ca4479f30 to worker[182] qsize 0
2022-07-25 23:27:46,883 DEBUG awx.main.dispatch task 53f0886d-1db8-4308-8c17-a75ca4479f30 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:27:46,886 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:27:46,945 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:27:47,018 DEBUG awx.main.dispatch task 53f0886d-1db8-4308-8c17-a75ca4479f30 is finished
RESULT 2
OKREADY
2022-07-25 23:28:06,867 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(acc9e125-6d85-45a9-aae2-fabd9105c768, queue=awx_private_queue)
[2022-07-25 23:28:06,867: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(acc9e125-6d85-45a9-aae2-fabd9105c768, queue=awx_private_queue)
2022-07-25 23:28:06,885 DEBUG awx.main.dispatch delivered acc9e125-6d85-45a9-aae2-fabd9105c768 to worker[181] qsize 0
2022-07-25 23:28:06,887 DEBUG awx.main.dispatch task acc9e125-6d85-45a9-aae2-fabd9105c768 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:28:06,889 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:28:06,959 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:28:07,044 DEBUG awx.main.dispatch task acc9e125-6d85-45a9-aae2-fabd9105c768 is finished
2022-07-25 23:28:16,785 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ff8b6a5d-0f50-4b9f-850a-72a40ff5cdc0, queue=awx_private_queue)
[2022-07-25 23:28:16,785: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ff8b6a5d-0f50-4b9f-850a-72a40ff5cdc0, queue=awx_private_queue)
2022-07-25 23:28:16,802 DEBUG awx.main.dispatch delivered ff8b6a5d-0f50-4b9f-850a-72a40ff5cdc0 to worker[180] qsize 0
2022-07-25 23:28:16,805 DEBUG awx.main.dispatch task ff8b6a5d-0f50-4b9f-850a-72a40ff5cdc0 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:28:16,822 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:28:16,826 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:27:46.825281+00:00
2022-07-25 23:28:16,851 DEBUG awx.main.dispatch task ff8b6a5d-0f50-4b9f-850a-72a40ff5cdc0 is finished
2022-07-25 23:28:26,868 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(999b6186-e0ea-46fe-be4d-5307768afe69, queue=awx_private_queue)
[2022-07-25 23:28:26,868: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(999b6186-e0ea-46fe-be4d-5307768afe69, queue=awx_private_queue)
2022-07-25 23:28:26,886 DEBUG awx.main.dispatch delivered 999b6186-e0ea-46fe-be4d-5307768afe69 to worker[182] qsize 0
2022-07-25 23:28:26,888 DEBUG awx.main.dispatch task 999b6186-e0ea-46fe-be4d-5307768afe69 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:28:26,890 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:28:26,936 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:28:27,006 DEBUG awx.main.dispatch task 999b6186-e0ea-46fe-be4d-5307768afe69 is finished
2022-07-25 23:28:46,474 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(88adf520-c497-4f38-ae93-ad469ade3561, queue=onap-awx-0)
[2022-07-25 23:28:46,474: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(88adf520-c497-4f38-ae93-ad469ade3561, queue=onap-awx-0)
2022-07-25 23:28:46,771 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3e9e19e0-e8e8-4ac2-9dc2-680be3ce7d89, queue=onap-awx-0)
[2022-07-25 23:28:46,771: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3e9e19e0-e8e8-4ac2-9dc2-680be3ce7d89, queue=onap-awx-0)
2022-07-25 23:28:46,788 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(44d24a1b-da95-4b67-a2f9-eeaba04f88f3, queue=awx_private_queue)
[2022-07-25 23:28:46,788: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(44d24a1b-da95-4b67-a2f9-eeaba04f88f3, queue=awx_private_queue)
2022-07-25 23:28:46,843 DEBUG awx.main.dispatch delivered 88adf520-c497-4f38-ae93-ad469ade3561 to worker[180] qsize 0
2022-07-25 23:28:46,846 DEBUG awx.main.dispatch delivered 3e9e19e0-e8e8-4ac2-9dc2-680be3ce7d89 to worker[182] qsize 0
2022-07-25 23:28:46,845 DEBUG awx.main.dispatch task 88adf520-c497-4f38-ae93-ad469ade3561 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:28:46,848 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:28:46,847 DEBUG awx.main.dispatch delivered 44d24a1b-da95-4b67-a2f9-eeaba04f88f3 to worker[183] qsize 0
2022-07-25 23:28:46,850 DEBUG awx.main.dispatch task 44d24a1b-da95-4b67-a2f9-eeaba04f88f3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:28:46,847 DEBUG awx.main.dispatch task 3e9e19e0-e8e8-4ac2-9dc2-680be3ce7d89 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:28:46,868 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c794758e-00db-4925-8a48-b7c3d11eb962, queue=awx_private_queue)
[2022-07-25 23:28:46,868: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c794758e-00db-4925-8a48-b7c3d11eb962, queue=awx_private_queue)
2022-07-25 23:28:46,885 DEBUG awx.main.dispatch delivered c794758e-00db-4925-8a48-b7c3d11eb962 to worker[181] qsize 0
2022-07-25 23:28:46,889 DEBUG awx.main.dispatch task c794758e-00db-4925-8a48-b7c3d11eb962 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:28:46,891 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:28:46,952 DEBUG awx.main.dispatch task 3e9e19e0-e8e8-4ac2-9dc2-680be3ce7d89 is finished
2022-07-25 23:28:46,969 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:28:46,981 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:28:16.822974+00:00
2022-07-25 23:28:46,989 DEBUG awx.main.dispatch task 88adf520-c497-4f38-ae93-ad469ade3561 is finished
2022-07-25 23:28:47,003 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:28:47,067 DEBUG awx.main.dispatch task 44d24a1b-da95-4b67-a2f9-eeaba04f88f3 is finished
2022-07-25 23:28:47,158 DEBUG awx.main.dispatch task c794758e-00db-4925-8a48-b7c3d11eb962 is finished
RESULT 2
OKREADY
2022-07-25 23:29:06,879 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(97d66990-f168-4219-9cc8-570fe0597598, queue=awx_private_queue)
[2022-07-25 23:29:06,879: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(97d66990-f168-4219-9cc8-570fe0597598, queue=awx_private_queue)
2022-07-25 23:29:06,897 DEBUG awx.main.dispatch delivered 97d66990-f168-4219-9cc8-570fe0597598 to worker[181] qsize 0
2022-07-25 23:29:06,900 DEBUG awx.main.dispatch task 97d66990-f168-4219-9cc8-570fe0597598 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:29:06,902 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:29:06,986 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:29:07,104 DEBUG awx.main.dispatch task 97d66990-f168-4219-9cc8-570fe0597598 is finished
2022-07-25 23:29:16,788 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(084dd320-58f0-488b-b3ef-fa2d831c288c, queue=awx_private_queue)
[2022-07-25 23:29:16,788: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(084dd320-58f0-488b-b3ef-fa2d831c288c, queue=awx_private_queue)
2022-07-25 23:29:16,806 DEBUG awx.main.dispatch delivered 084dd320-58f0-488b-b3ef-fa2d831c288c to worker[182] qsize 0
2022-07-25 23:29:16,809 DEBUG awx.main.dispatch task 084dd320-58f0-488b-b3ef-fa2d831c288c starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:29:16,832 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:29:16,838 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:28:46.969693+00:00
2022-07-25 23:29:16,873 DEBUG awx.main.dispatch task 084dd320-58f0-488b-b3ef-fa2d831c288c is finished
2022-07-25 23:29:26,881 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b2120eaa-d28b-4d3c-9a82-dfc37e157fc1, queue=awx_private_queue)
[2022-07-25 23:29:26,881: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b2120eaa-d28b-4d3c-9a82-dfc37e157fc1, queue=awx_private_queue)
2022-07-25 23:29:26,902 DEBUG awx.main.dispatch delivered b2120eaa-d28b-4d3c-9a82-dfc37e157fc1 to worker[181] qsize 0
2022-07-25 23:29:26,905 DEBUG awx.main.dispatch task b2120eaa-d28b-4d3c-9a82-dfc37e157fc1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:29:26,907 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:29:26,946 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:29:27,039 DEBUG awx.main.dispatch task b2120eaa-d28b-4d3c-9a82-dfc37e157fc1 is finished
2022-07-25 23:29:46,484 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b3df1d38-16bb-44ac-9837-b0671c12b520, queue=onap-awx-0)
[2022-07-25 23:29:46,484: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b3df1d38-16bb-44ac-9837-b0671c12b520, queue=onap-awx-0)
2022-07-25 23:29:46,765 DEBUG awx.main.dispatch delivered b3df1d38-16bb-44ac-9837-b0671c12b520 to worker[180] qsize 0
2022-07-25 23:29:46,768 DEBUG awx.main.dispatch task b3df1d38-16bb-44ac-9837-b0671c12b520 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:29:46,770 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:29:46,772 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(84db93d4-84d7-436a-a749-4ea319a472af, queue=onap-awx-0)
[2022-07-25 23:29:46,772: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(84db93d4-84d7-436a-a749-4ea319a472af, queue=onap-awx-0)
2022-07-25 23:29:46,791 DEBUG awx.main.dispatch delivered 84db93d4-84d7-436a-a749-4ea319a472af to worker[183] qsize 0
2022-07-25 23:29:46,793 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(119601bf-cf84-4db3-819d-6d7445fa78ba, queue=awx_private_queue)
[2022-07-25 23:29:46,793: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(119601bf-cf84-4db3-819d-6d7445fa78ba, queue=awx_private_queue)
2022-07-25 23:29:46,794 DEBUG awx.main.dispatch task 84db93d4-84d7-436a-a749-4ea319a472af starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:29:46,809 DEBUG awx.main.dispatch delivered 119601bf-cf84-4db3-819d-6d7445fa78ba to worker[181] qsize 0
2022-07-25 23:29:46,811 DEBUG awx.main.dispatch task 119601bf-cf84-4db3-819d-6d7445fa78ba starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:29:46,818 DEBUG awx.main.dispatch task b3df1d38-16bb-44ac-9837-b0671c12b520 is finished
2022-07-25 23:29:46,826 DEBUG awx.main.dispatch task 84db93d4-84d7-436a-a749-4ea319a472af is finished
2022-07-25 23:29:46,828 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:29:46,834 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:29:16.832582+00:00
2022-07-25 23:29:46,869 DEBUG awx.main.dispatch task 119601bf-cf84-4db3-819d-6d7445fa78ba is finished
2022-07-25 23:29:46,880 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(667f413e-7c35-44b5-b981-58878077da18, queue=awx_private_queue)
[2022-07-25 23:29:46,880: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(667f413e-7c35-44b5-b981-58878077da18, queue=awx_private_queue)
2022-07-25 23:29:46,897 DEBUG awx.main.dispatch delivered 667f413e-7c35-44b5-b981-58878077da18 to worker[183] qsize 0
2022-07-25 23:29:46,901 DEBUG awx.main.dispatch task 667f413e-7c35-44b5-b981-58878077da18 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:29:46,903 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:29:46,933 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:29:46,986 DEBUG awx.main.dispatch task 667f413e-7c35-44b5-b981-58878077da18 is finished
RESULT 2
OKREADY
2022-07-25 23:30:06,886 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6ddbdeee-d161-4ad4-a59f-a136c3795823, queue=awx_private_queue)
[2022-07-25 23:30:06,886: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6ddbdeee-d161-4ad4-a59f-a136c3795823, queue=awx_private_queue)
2022-07-25 23:30:06,908 DEBUG awx.main.dispatch delivered 6ddbdeee-d161-4ad4-a59f-a136c3795823 to worker[180] qsize 0
2022-07-25 23:30:06,911 DEBUG awx.main.dispatch task 6ddbdeee-d161-4ad4-a59f-a136c3795823 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:30:06,915 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:30:06,977 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:30:07,093 DEBUG awx.main.dispatch task 6ddbdeee-d161-4ad4-a59f-a136c3795823 is finished
2022-07-25 23:30:16,793 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7c2fea01-0db1-47cd-8e57-4d9157196115, queue=awx_private_queue)
[2022-07-25 23:30:16,793: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7c2fea01-0db1-47cd-8e57-4d9157196115, queue=awx_private_queue)
2022-07-25 23:30:16,812 DEBUG awx.main.dispatch delivered 7c2fea01-0db1-47cd-8e57-4d9157196115 to worker[182] qsize 0
2022-07-25 23:30:16,815 DEBUG awx.main.dispatch task 7c2fea01-0db1-47cd-8e57-4d9157196115 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:30:16,935 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:30:16,946 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:29:46.829114+00:00
2022-07-25 23:30:17,059 DEBUG awx.main.dispatch task 7c2fea01-0db1-47cd-8e57-4d9157196115 is finished
2022-07-25 23:30:26,885 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a545cbe2-eaf1-4767-83b7-bd1ca717ff3e, queue=awx_private_queue)
[2022-07-25 23:30:26,885: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a545cbe2-eaf1-4767-83b7-bd1ca717ff3e, queue=awx_private_queue)
2022-07-25 23:30:26,903 DEBUG awx.main.dispatch delivered a545cbe2-eaf1-4767-83b7-bd1ca717ff3e to worker[180] qsize 0
2022-07-25 23:30:26,905 DEBUG awx.main.dispatch task a545cbe2-eaf1-4767-83b7-bd1ca717ff3e starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:30:26,907 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:30:26,992 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:30:27,147 DEBUG awx.main.dispatch task a545cbe2-eaf1-4767-83b7-bd1ca717ff3e is finished
2022-07-25 23:30:46,483 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(3014ec8a-e1a5-4900-9695-69dea8e90a26, queue=onap-awx-0)
[2022-07-25 23:30:46,483: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(3014ec8a-e1a5-4900-9695-69dea8e90a26, queue=onap-awx-0)
2022-07-25 23:30:46,581 DEBUG awx.main.dispatch delivered 3014ec8a-e1a5-4900-9695-69dea8e90a26 to worker[181] qsize 0
2022-07-25 23:30:46,585 DEBUG awx.main.dispatch task 3014ec8a-e1a5-4900-9695-69dea8e90a26 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:30:46,851 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:30:46,939 DEBUG awx.main.dispatch task 3014ec8a-e1a5-4900-9695-69dea8e90a26 is finished
2022-07-25 23:30:46,771 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d5dfcaac-7a28-4012-93b5-4b866e6ba104, queue=onap-awx-0)
[2022-07-25 23:30:46,771: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d5dfcaac-7a28-4012-93b5-4b866e6ba104, queue=onap-awx-0)
2022-07-25 23:30:47,085 DEBUG awx.main.dispatch delivered d5dfcaac-7a28-4012-93b5-4b866e6ba104 to worker[182] qsize 0
2022-07-25 23:30:47,088 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c14ae36d-c7d8-4d67-b674-24fffb075171, queue=awx_private_queue)
[2022-07-25 23:30:47,088: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c14ae36d-c7d8-4d67-b674-24fffb075171, queue=awx_private_queue)
2022-07-25 23:30:47,089 DEBUG awx.main.dispatch task d5dfcaac-7a28-4012-93b5-4b866e6ba104 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:30:47,106 DEBUG awx.main.dispatch delivered c14ae36d-c7d8-4d67-b674-24fffb075171 to worker[181] qsize 0
2022-07-25 23:30:47,108 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0728cb17-5b1a-48c8-be1e-ced311b29580, queue=awx_private_queue)
[2022-07-25 23:30:47,108: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0728cb17-5b1a-48c8-be1e-ced311b29580, queue=awx_private_queue)
2022-07-25 23:30:47,109 DEBUG awx.main.dispatch task c14ae36d-c7d8-4d67-b674-24fffb075171 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:30:47,126 DEBUG awx.main.dispatch delivered 0728cb17-5b1a-48c8-be1e-ced311b29580 to worker[183] qsize 0
2022-07-25 23:30:47,130 DEBUG awx.main.dispatch task 0728cb17-5b1a-48c8-be1e-ced311b29580 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:30:47,132 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:30:47,207 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:30:47,207 DEBUG awx.main.dispatch task d5dfcaac-7a28-4012-93b5-4b866e6ba104 is finished
2022-07-25 23:30:47,215 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:30:16.935469+00:00
2022-07-25 23:30:47,238 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:30:47,478 DEBUG awx.main.dispatch task c14ae36d-c7d8-4d67-b674-24fffb075171 is finished
2022-07-25 23:30:47,532 DEBUG awx.main.dispatch task 0728cb17-5b1a-48c8-be1e-ced311b29580 is finished
RESULT 2
OKREADY
2022-07-25 23:31:07,115 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3a3f9595-bfb8-4258-b1ef-79186af19c3d, queue=awx_private_queue)
[2022-07-25 23:31:07,115: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3a3f9595-bfb8-4258-b1ef-79186af19c3d, queue=awx_private_queue)
2022-07-25 23:31:07,135 DEBUG awx.main.dispatch delivered 3a3f9595-bfb8-4258-b1ef-79186af19c3d to worker[182] qsize 0
2022-07-25 23:31:07,139 DEBUG awx.main.dispatch task 3a3f9595-bfb8-4258-b1ef-79186af19c3d starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:31:07,142 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:31:07,303 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:31:07,486 DEBUG awx.main.dispatch task 3a3f9595-bfb8-4258-b1ef-79186af19c3d is finished
2022-07-25 23:31:17,087 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e644adde-7b8b-4565-be22-4b5471abe170, queue=awx_private_queue)
[2022-07-25 23:31:17,087: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e644adde-7b8b-4565-be22-4b5471abe170, queue=awx_private_queue)
2022-07-25 23:31:17,106 DEBUG awx.main.dispatch delivered e644adde-7b8b-4565-be22-4b5471abe170 to worker[180] qsize 0
2022-07-25 23:31:17,108 DEBUG awx.main.dispatch task e644adde-7b8b-4565-be22-4b5471abe170 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:31:17,198 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:31:17,236 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:30:47.208448+00:00
2022-07-25 23:31:17,332 DEBUG awx.main.dispatch task e644adde-7b8b-4565-be22-4b5471abe170 is finished
2022-07-25 23:31:27,115 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8ef1b503-f8d6-46d2-b23f-8bb552054022, queue=awx_private_queue)
[2022-07-25 23:31:27,115: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8ef1b503-f8d6-46d2-b23f-8bb552054022, queue=awx_private_queue)
2022-07-25 23:31:27,134 DEBUG awx.main.dispatch delivered 8ef1b503-f8d6-46d2-b23f-8bb552054022 to worker[183] qsize 0
2022-07-25 23:31:27,137 DEBUG awx.main.dispatch task 8ef1b503-f8d6-46d2-b23f-8bb552054022 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:31:27,138 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:31:27,310 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:31:27,886 DEBUG awx.main.dispatch task 8ef1b503-f8d6-46d2-b23f-8bb552054022 is finished
2022-07-25 23:31:46,494 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(bd76a983-234f-4689-bac0-3f5edc2ee5dc, queue=onap-awx-0)
[2022-07-25 23:31:46,494: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(bd76a983-234f-4689-bac0-3f5edc2ee5dc, queue=onap-awx-0)
2022-07-25 23:31:46,597 DEBUG awx.main.dispatch delivered bd76a983-234f-4689-bac0-3f5edc2ee5dc to worker[180] qsize 0
2022-07-25 23:31:46,610 DEBUG awx.main.dispatch task bd76a983-234f-4689-bac0-3f5edc2ee5dc starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:31:46,612 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:31:46,617 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:31:46,626 DEBUG awx.main.dispatch task bd76a983-234f-4689-bac0-3f5edc2ee5dc is finished
2022-07-25 23:31:46,771 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(4bddb954-337e-4997-a3b3-c852404f71e1, queue=onap-awx-0)
[2022-07-25 23:31:46,771: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(4bddb954-337e-4997-a3b3-c852404f71e1, queue=onap-awx-0)
2022-07-25 23:31:46,795 DEBUG awx.main.dispatch delivered 4bddb954-337e-4997-a3b3-c852404f71e1 to worker[181] qsize 0
2022-07-25 23:31:46,797 DEBUG awx.main.dispatch task 4bddb954-337e-4997-a3b3-c852404f71e1 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:31:46,814 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:31:46,817 DEBUG awx.main.dispatch task 4bddb954-337e-4997-a3b3-c852404f71e1 is finished
2022-07-25 23:31:47,088 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7012d23a-8cbd-42c6-b3b1-3823226aea13, queue=awx_private_queue)
[2022-07-25 23:31:47,088: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7012d23a-8cbd-42c6-b3b1-3823226aea13, queue=awx_private_queue)
2022-07-25 23:31:47,115 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d66b10a0-e5a8-4ddb-9898-425990ad9d51, queue=awx_private_queue)
[2022-07-25 23:31:47,115: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d66b10a0-e5a8-4ddb-9898-425990ad9d51, queue=awx_private_queue)
2022-07-25 23:31:47,105 DEBUG awx.main.dispatch delivered 7012d23a-8cbd-42c6-b3b1-3823226aea13 to worker[183] qsize 0
2022-07-25 23:31:47,356 DEBUG awx.main.dispatch task 7012d23a-8cbd-42c6-b3b1-3823226aea13 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:31:47,357 DEBUG awx.main.dispatch delivered d66b10a0-e5a8-4ddb-9898-425990ad9d51 to worker[182] qsize 0
2022-07-25 23:31:47,362 DEBUG awx.main.dispatch task d66b10a0-e5a8-4ddb-9898-425990ad9d51 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:31:47,364 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:31:47,362 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:31:47,370 DEBUG awx.main.dispatch task 7012d23a-8cbd-42c6-b3b1-3823226aea13 is finished
2022-07-25 23:31:47,367 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:31:47,372 DEBUG awx.main.dispatch task d66b10a0-e5a8-4ddb-9898-425990ad9d51 is finished
RESULT 2
OKREADY
2022-07-25 23:32:07,135 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0213fefe-13a8-4c9b-974d-03959186ac8b, queue=awx_private_queue)
[2022-07-25 23:32:07,135: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0213fefe-13a8-4c9b-974d-03959186ac8b, queue=awx_private_queue)
2022-07-25 23:32:07,154 DEBUG awx.main.dispatch delivered 0213fefe-13a8-4c9b-974d-03959186ac8b to worker[180] qsize 0
2022-07-25 23:32:07,157 DEBUG awx.main.dispatch task 0213fefe-13a8-4c9b-974d-03959186ac8b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:32:07,160 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:32:07,271 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:32:07,497 DEBUG awx.main.dispatch task 0213fefe-13a8-4c9b-974d-03959186ac8b is finished
2022-07-25 23:32:17,088 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(722c586a-c83b-473e-8e19-a20df7aa9d18, queue=awx_private_queue)
[2022-07-25 23:32:17,088: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(722c586a-c83b-473e-8e19-a20df7aa9d18, queue=awx_private_queue)
2022-07-25 23:32:17,105 DEBUG awx.main.dispatch delivered 722c586a-c83b-473e-8e19-a20df7aa9d18 to worker[181] qsize 0
2022-07-25 23:32:17,108 DEBUG awx.main.dispatch task 722c586a-c83b-473e-8e19-a20df7aa9d18 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:32:17,131 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:32:17,136 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:31:17.198655+00:00
2022-07-25 23:32:17,175 DEBUG awx.main.dispatch task 722c586a-c83b-473e-8e19-a20df7aa9d18 is finished
2022-07-25 23:32:27,136 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(08a76314-f587-4850-8438-1aafda29135c, queue=awx_private_queue)
[2022-07-25 23:32:27,136: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(08a76314-f587-4850-8438-1aafda29135c, queue=awx_private_queue)
2022-07-25 23:32:27,159 DEBUG awx.main.dispatch delivered 08a76314-f587-4850-8438-1aafda29135c to worker[180] qsize 0
2022-07-25 23:32:27,162 DEBUG awx.main.dispatch task 08a76314-f587-4850-8438-1aafda29135c starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:32:27,166 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:32:27,381 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:32:27,819 DEBUG awx.main.dispatch task 08a76314-f587-4850-8438-1aafda29135c is finished
2022-07-25 23:32:46,503 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e34d2541-bda4-41d3-9948-bbeefb1de6b4, queue=onap-awx-0)
[2022-07-25 23:32:46,503: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e34d2541-bda4-41d3-9948-bbeefb1de6b4, queue=onap-awx-0)
2022-07-25 23:32:46,562 DEBUG awx.main.dispatch delivered e34d2541-bda4-41d3-9948-bbeefb1de6b4 to worker[181] qsize 0
2022-07-25 23:32:46,565 DEBUG awx.main.dispatch task e34d2541-bda4-41d3-9948-bbeefb1de6b4 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:32:46,568 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:32:46,690 DEBUG awx.main.dispatch task e34d2541-bda4-41d3-9948-bbeefb1de6b4 is finished
2022-07-25 23:32:46,771 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8e007d8f-6bcb-41a9-bed6-120df9082220, queue=onap-awx-0)
[2022-07-25 23:32:46,771: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8e007d8f-6bcb-41a9-bed6-120df9082220, queue=onap-awx-0)
2022-07-25 23:32:46,788 DEBUG awx.main.dispatch delivered 8e007d8f-6bcb-41a9-bed6-120df9082220 to worker[180] qsize 0
2022-07-25 23:32:46,791 DEBUG awx.main.dispatch task 8e007d8f-6bcb-41a9-bed6-120df9082220 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:32:46,833 DEBUG awx.main.dispatch task 8e007d8f-6bcb-41a9-bed6-120df9082220 is finished
2022-07-25 23:32:47,088 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(03c3ebcf-7f87-4ef5-8f63-8dd9c2924da1, queue=awx_private_queue)
[2022-07-25 23:32:47,088: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(03c3ebcf-7f87-4ef5-8f63-8dd9c2924da1, queue=awx_private_queue)
2022-07-25 23:32:47,105 DEBUG awx.main.dispatch delivered 03c3ebcf-7f87-4ef5-8f63-8dd9c2924da1 to worker[180] qsize 0
2022-07-25 23:32:47,109 DEBUG awx.main.dispatch task 03c3ebcf-7f87-4ef5-8f63-8dd9c2924da1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:32:47,136 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(48cfa871-bf01-4a41-aae9-c15ded013c77, queue=awx_private_queue)
[2022-07-25 23:32:47,136: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(48cfa871-bf01-4a41-aae9-c15ded013c77, queue=awx_private_queue)
2022-07-25 23:32:47,153 DEBUG awx.main.dispatch delivered 48cfa871-bf01-4a41-aae9-c15ded013c77 to worker[183] qsize 0
2022-07-25 23:32:47,156 DEBUG awx.main.dispatch task 48cfa871-bf01-4a41-aae9-c15ded013c77 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:32:47,159 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:32:47,183 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:32:47,200 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:32:17.131367+00:00
2022-07-25 23:32:47,275 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:32:47,315 DEBUG awx.main.dispatch task 03c3ebcf-7f87-4ef5-8f63-8dd9c2924da1 is finished
2022-07-25 23:32:47,472 DEBUG awx.main.dispatch task 48cfa871-bf01-4a41-aae9-c15ded013c77 is finished
RESULT 2
OKREADY
2022-07-25 23:33:07,143 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e5f34422-5075-4b4f-afab-63e94fe3ca27, queue=awx_private_queue)
[2022-07-25 23:33:07,143: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e5f34422-5075-4b4f-afab-63e94fe3ca27, queue=awx_private_queue)
2022-07-25 23:33:07,163 DEBUG awx.main.dispatch delivered e5f34422-5075-4b4f-afab-63e94fe3ca27 to worker[180] qsize 0
2022-07-25 23:33:07,166 DEBUG awx.main.dispatch task e5f34422-5075-4b4f-afab-63e94fe3ca27 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:33:07,168 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:33:07,266 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:33:07,404 DEBUG awx.main.dispatch task e5f34422-5075-4b4f-afab-63e94fe3ca27 is finished
2022-07-25 23:33:17,089 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b2517b80-8a69-4c7b-a6ce-ac3107ba0a31, queue=awx_private_queue)
[2022-07-25 23:33:17,089: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b2517b80-8a69-4c7b-a6ce-ac3107ba0a31, queue=awx_private_queue)
2022-07-25 23:33:17,116 DEBUG awx.main.dispatch delivered b2517b80-8a69-4c7b-a6ce-ac3107ba0a31 to worker[182] qsize 0
2022-07-25 23:33:17,120 DEBUG awx.main.dispatch task b2517b80-8a69-4c7b-a6ce-ac3107ba0a31 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:33:17,165 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:33:17,177 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:32:47.183422+00:00
2022-07-25 23:33:17,225 DEBUG awx.main.dispatch task b2517b80-8a69-4c7b-a6ce-ac3107ba0a31 is finished
2022-07-25 23:33:27,143 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8c24027f-1a62-4d8c-b3be-5ffea3130151, queue=awx_private_queue)
[2022-07-25 23:33:27,143: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8c24027f-1a62-4d8c-b3be-5ffea3130151, queue=awx_private_queue)
2022-07-25 23:33:27,162 DEBUG awx.main.dispatch delivered 8c24027f-1a62-4d8c-b3be-5ffea3130151 to worker[183] qsize 0
2022-07-25 23:33:27,165 DEBUG awx.main.dispatch task 8c24027f-1a62-4d8c-b3be-5ffea3130151 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:33:27,167 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:33:27,417 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:33:27,652 DEBUG awx.main.dispatch task 8c24027f-1a62-4d8c-b3be-5ffea3130151 is finished
2022-07-25 23:33:46,511 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(dc63c3a8-ca92-4794-97a8-638b17872b4b, queue=onap-awx-0)
[2022-07-25 23:33:46,511: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(dc63c3a8-ca92-4794-97a8-638b17872b4b, queue=onap-awx-0)
2022-07-25 23:33:46,653 DEBUG awx.main.dispatch delivered dc63c3a8-ca92-4794-97a8-638b17872b4b to worker[182] qsize 0
2022-07-25 23:33:46,656 DEBUG awx.main.dispatch task dc63c3a8-ca92-4794-97a8-638b17872b4b starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:33:46,658 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:33:46,771 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(6cab9521-64f7-480f-9753-a5c8f6213fab, queue=onap-awx-0)
[2022-07-25 23:33:46,771: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(6cab9521-64f7-480f-9753-a5c8f6213fab, queue=onap-awx-0)
2022-07-25 23:33:46,787 DEBUG awx.main.dispatch delivered 6cab9521-64f7-480f-9753-a5c8f6213fab to worker[183] qsize 0
2022-07-25 23:33:46,790 DEBUG awx.main.dispatch task 6cab9521-64f7-480f-9753-a5c8f6213fab starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:33:47,043 DEBUG awx.main.dispatch task 6cab9521-64f7-480f-9753-a5c8f6213fab is finished
2022-07-25 23:33:47,089 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(31d27ed5-85c6-40b2-86d8-3d64b12bfec9, queue=awx_private_queue)
[2022-07-25 23:33:47,089: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(31d27ed5-85c6-40b2-86d8-3d64b12bfec9, queue=awx_private_queue)
2022-07-25 23:33:47,106 DEBUG awx.main.dispatch delivered 31d27ed5-85c6-40b2-86d8-3d64b12bfec9 to worker[183] qsize 0
2022-07-25 23:33:47,109 DEBUG awx.main.dispatch task 31d27ed5-85c6-40b2-86d8-3d64b12bfec9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:33:47,143 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cee1d987-a394-40d5-b7d1-1ef3475a7532, queue=awx_private_queue)
[2022-07-25 23:33:47,143: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cee1d987-a394-40d5-b7d1-1ef3475a7532, queue=awx_private_queue)
2022-07-25 23:33:47,159 DEBUG awx.main.dispatch delivered cee1d987-a394-40d5-b7d1-1ef3475a7532 to worker[180] qsize 0
2022-07-25 23:33:47,161 DEBUG awx.main.dispatch task cee1d987-a394-40d5-b7d1-1ef3475a7532 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:33:47,164 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:33:47,258 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:33:47,310 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:33:17.165966+00:00
2022-07-25 23:33:47,409 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:33:47,384 DEBUG awx.main.dispatch task dc63c3a8-ca92-4794-97a8-638b17872b4b is finished
2022-07-25 23:33:47,648 DEBUG awx.main.dispatch task cee1d987-a394-40d5-b7d1-1ef3475a7532 is finished
2022-07-25 23:33:47,487 DEBUG awx.main.dispatch task 31d27ed5-85c6-40b2-86d8-3d64b12bfec9 is finished
RESULT 2
OKREADY
2022-07-25 23:34:07,151 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ff3f7d60-b7a8-41a7-ada0-a0f57ef7377e, queue=awx_private_queue)
[2022-07-25 23:34:07,151: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ff3f7d60-b7a8-41a7-ada0-a0f57ef7377e, queue=awx_private_queue)
2022-07-25 23:34:07,168 DEBUG awx.main.dispatch delivered ff3f7d60-b7a8-41a7-ada0-a0f57ef7377e to worker[181] qsize 0
2022-07-25 23:34:07,173 DEBUG awx.main.dispatch task ff3f7d60-b7a8-41a7-ada0-a0f57ef7377e starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:34:07,177 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:34:07,281 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:34:07,386 DEBUG awx.main.dispatch task ff3f7d60-b7a8-41a7-ada0-a0f57ef7377e is finished
2022-07-25 23:34:17,089 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bbb6457a-8f84-43ff-88dd-5c4619f82c96, queue=awx_private_queue)
[2022-07-25 23:34:17,089: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bbb6457a-8f84-43ff-88dd-5c4619f82c96, queue=awx_private_queue)
2022-07-25 23:34:17,108 DEBUG awx.main.dispatch delivered bbb6457a-8f84-43ff-88dd-5c4619f82c96 to worker[183] qsize 0
2022-07-25 23:34:17,110 DEBUG awx.main.dispatch task bbb6457a-8f84-43ff-88dd-5c4619f82c96 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:34:17,229 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:34:17,264 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:33:47.259010+00:00
2022-07-25 23:34:17,425 DEBUG awx.main.dispatch task bbb6457a-8f84-43ff-88dd-5c4619f82c96 is finished
2022-07-25 23:34:27,152 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b86ed677-6aee-4b61-a469-e5c518773e69, queue=awx_private_queue)
[2022-07-25 23:34:27,152: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b86ed677-6aee-4b61-a469-e5c518773e69, queue=awx_private_queue)
2022-07-25 23:34:27,173 DEBUG awx.main.dispatch delivered b86ed677-6aee-4b61-a469-e5c518773e69 to worker[183] qsize 0
2022-07-25 23:34:27,176 DEBUG awx.main.dispatch task b86ed677-6aee-4b61-a469-e5c518773e69 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:34:27,177 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:34:27,291 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:34:27,466 DEBUG awx.main.dispatch task b86ed677-6aee-4b61-a469-e5c518773e69 is finished
2022-07-25 23:34:46,521 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e875b1bb-dc3a-4c26-af71-017c3b7caa26, queue=onap-awx-0)
[2022-07-25 23:34:46,521: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e875b1bb-dc3a-4c26-af71-017c3b7caa26, queue=onap-awx-0)
2022-07-25 23:34:46,603 DEBUG awx.main.dispatch delivered e875b1bb-dc3a-4c26-af71-017c3b7caa26 to worker[181] qsize 0
2022-07-25 23:34:46,605 DEBUG awx.main.dispatch task e875b1bb-dc3a-4c26-af71-017c3b7caa26 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:34:46,607 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:34:46,737 DEBUG awx.main.dispatch task e875b1bb-dc3a-4c26-af71-017c3b7caa26 is finished
2022-07-25 23:34:46,771 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(dab13901-24e0-4e72-83b1-dfa35c0349e6, queue=onap-awx-0)
[2022-07-25 23:34:46,771: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(dab13901-24e0-4e72-83b1-dfa35c0349e6, queue=onap-awx-0)
2022-07-25 23:34:46,788 DEBUG awx.main.dispatch delivered dab13901-24e0-4e72-83b1-dfa35c0349e6 to worker[182] qsize 0
2022-07-25 23:34:46,790 DEBUG awx.main.dispatch task dab13901-24e0-4e72-83b1-dfa35c0349e6 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:34:46,879 DEBUG awx.main.dispatch task dab13901-24e0-4e72-83b1-dfa35c0349e6 is finished
2022-07-25 23:34:47,089 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(85f2f108-f1f0-4fea-a746-4c8250b1ea84, queue=awx_private_queue)
[2022-07-25 23:34:47,089: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(85f2f108-f1f0-4fea-a746-4c8250b1ea84, queue=awx_private_queue)
2022-07-25 23:34:47,105 DEBUG awx.main.dispatch delivered 85f2f108-f1f0-4fea-a746-4c8250b1ea84 to worker[183] qsize 0
2022-07-25 23:34:47,108 DEBUG awx.main.dispatch task 85f2f108-f1f0-4fea-a746-4c8250b1ea84 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:34:47,152 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(da0b3e2b-b6ec-4ee2-ab94-2e79ef7b0208, queue=awx_private_queue)
[2022-07-25 23:34:47,152: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(da0b3e2b-b6ec-4ee2-ab94-2e79ef7b0208, queue=awx_private_queue)
2022-07-25 23:34:47,169 DEBUG awx.main.dispatch delivered da0b3e2b-b6ec-4ee2-ab94-2e79ef7b0208 to worker[181] qsize 0
2022-07-25 23:34:47,173 DEBUG awx.main.dispatch task da0b3e2b-b6ec-4ee2-ab94-2e79ef7b0208 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:34:47,175 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:34:47,206 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:34:47,213 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:34:17.229944+00:00
2022-07-25 23:34:47,239 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:34:47,300 DEBUG awx.main.dispatch task 85f2f108-f1f0-4fea-a746-4c8250b1ea84 is finished
2022-07-25 23:34:47,361 DEBUG awx.main.dispatch task da0b3e2b-b6ec-4ee2-ab94-2e79ef7b0208 is finished
RESULT 2
OKREADY
2022-07-25 23:35:07,170 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(27ee1e2f-fee3-4b99-b751-6a5410ab846c, queue=awx_private_queue)
[2022-07-25 23:35:07,170: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(27ee1e2f-fee3-4b99-b751-6a5410ab846c, queue=awx_private_queue)
2022-07-25 23:35:07,292 DEBUG awx.main.dispatch delivered 27ee1e2f-fee3-4b99-b751-6a5410ab846c to worker[180] qsize 0
2022-07-25 23:35:07,295 DEBUG awx.main.dispatch task 27ee1e2f-fee3-4b99-b751-6a5410ab846c starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:35:07,573 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:35:07,592 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:35:07,678 DEBUG awx.main.dispatch task 27ee1e2f-fee3-4b99-b751-6a5410ab846c is finished
2022-07-25 23:35:17,090 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(46ca9e11-e5d7-4c32-9435-cb12ccae2b9a, queue=awx_private_queue)
[2022-07-25 23:35:17,090: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(46ca9e11-e5d7-4c32-9435-cb12ccae2b9a, queue=awx_private_queue)
2022-07-25 23:35:17,112 DEBUG awx.main.dispatch delivered 46ca9e11-e5d7-4c32-9435-cb12ccae2b9a to worker[182] qsize 0
2022-07-25 23:35:17,115 DEBUG awx.main.dispatch task 46ca9e11-e5d7-4c32-9435-cb12ccae2b9a starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:35:17,244 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:35:17,272 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:34:47.206229+00:00
2022-07-25 23:35:17,493 DEBUG awx.main.dispatch task 46ca9e11-e5d7-4c32-9435-cb12ccae2b9a is finished
2022-07-25 23:35:27,171 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fde6a728-f50c-43c0-a911-ecbf4d9151e4, queue=awx_private_queue)
[2022-07-25 23:35:27,171: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fde6a728-f50c-43c0-a911-ecbf4d9151e4, queue=awx_private_queue)
2022-07-25 23:35:27,191 DEBUG awx.main.dispatch delivered fde6a728-f50c-43c0-a911-ecbf4d9151e4 to worker[183] qsize 0
2022-07-25 23:35:27,196 DEBUG awx.main.dispatch task fde6a728-f50c-43c0-a911-ecbf4d9151e4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:35:27,201 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:35:27,365 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:35:27,684 DEBUG awx.main.dispatch task fde6a728-f50c-43c0-a911-ecbf4d9151e4 is finished
2022-07-25 23:35:46,531 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8dee6bec-5d0e-4606-b458-0899ea3ff3e2, queue=onap-awx-0)
[2022-07-25 23:35:46,531: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8dee6bec-5d0e-4606-b458-0899ea3ff3e2, queue=onap-awx-0)
2022-07-25 23:35:46,584 DEBUG awx.main.dispatch delivered 8dee6bec-5d0e-4606-b458-0899ea3ff3e2 to worker[183] qsize 0
2022-07-25 23:35:46,587 DEBUG awx.main.dispatch task 8dee6bec-5d0e-4606-b458-0899ea3ff3e2 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:35:46,588 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:35:46,656 DEBUG awx.main.dispatch task 8dee6bec-5d0e-4606-b458-0899ea3ff3e2 is finished
2022-07-25 23:35:46,771 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c75075e0-24a4-4356-b742-2329eb774def, queue=onap-awx-0)
[2022-07-25 23:35:46,771: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c75075e0-24a4-4356-b742-2329eb774def, queue=onap-awx-0)
2022-07-25 23:35:46,793 DEBUG awx.main.dispatch delivered c75075e0-24a4-4356-b742-2329eb774def to worker[182] qsize 0
2022-07-25 23:35:46,795 DEBUG awx.main.dispatch task c75075e0-24a4-4356-b742-2329eb774def starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:35:46,879 DEBUG awx.main.dispatch task c75075e0-24a4-4356-b742-2329eb774def is finished
2022-07-25 23:35:47,089 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a38821d4-2617-4dc5-bb6e-e9036d36a895, queue=awx_private_queue)
[2022-07-25 23:35:47,089: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a38821d4-2617-4dc5-bb6e-e9036d36a895, queue=awx_private_queue)
2022-07-25 23:35:47,108 DEBUG awx.main.dispatch delivered a38821d4-2617-4dc5-bb6e-e9036d36a895 to worker[182] qsize 0
2022-07-25 23:35:47,111 DEBUG awx.main.dispatch task a38821d4-2617-4dc5-bb6e-e9036d36a895 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:35:47,171 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9b9c04da-3230-4b9e-a6c3-cab9ec293f55, queue=awx_private_queue)
[2022-07-25 23:35:47,171: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9b9c04da-3230-4b9e-a6c3-cab9ec293f55, queue=awx_private_queue)
2022-07-25 23:35:47,178 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:35:47,199 DEBUG awx.main.dispatch delivered 9b9c04da-3230-4b9e-a6c3-cab9ec293f55 to worker[183] qsize 0
2022-07-25 23:35:47,204 DEBUG awx.main.dispatch task 9b9c04da-3230-4b9e-a6c3-cab9ec293f55 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:35:47,206 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:35:47,226 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:35:17.245204+00:00
2022-07-25 23:35:47,372 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:35:47,484 DEBUG awx.main.dispatch task a38821d4-2617-4dc5-bb6e-e9036d36a895 is finished
2022-07-25 23:35:47,632 DEBUG awx.main.dispatch task 9b9c04da-3230-4b9e-a6c3-cab9ec293f55 is finished
RESULT 2
OKREADY
2022-07-25 23:36:07,174 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3975d6cd-6c20-4bed-83d6-e87246892c78, queue=awx_private_queue)
[2022-07-25 23:36:07,174: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3975d6cd-6c20-4bed-83d6-e87246892c78, queue=awx_private_queue)
2022-07-25 23:36:07,191 DEBUG awx.main.dispatch delivered 3975d6cd-6c20-4bed-83d6-e87246892c78 to worker[182] qsize 0
2022-07-25 23:36:07,193 DEBUG awx.main.dispatch task 3975d6cd-6c20-4bed-83d6-e87246892c78 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:36:07,194 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:36:07,452 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:36:07,639 DEBUG awx.main.dispatch task 3975d6cd-6c20-4bed-83d6-e87246892c78 is finished
2022-07-25 23:36:17,089 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(cb8e6d49-2aae-4942-9364-29a3b9327807, queue=awx_private_queue)
[2022-07-25 23:36:17,089: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(cb8e6d49-2aae-4942-9364-29a3b9327807, queue=awx_private_queue)
2022-07-25 23:36:17,107 DEBUG awx.main.dispatch delivered cb8e6d49-2aae-4942-9364-29a3b9327807 to worker[180] qsize 0
2022-07-25 23:36:17,110 DEBUG awx.main.dispatch task cb8e6d49-2aae-4942-9364-29a3b9327807 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:36:17,455 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:36:17,465 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:35:47.178713+00:00
2022-07-25 23:36:17,624 DEBUG awx.main.dispatch task cb8e6d49-2aae-4942-9364-29a3b9327807 is finished
2022-07-25 23:36:27,174 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2dc410ab-b90e-47bb-bef8-4b747c407432, queue=awx_private_queue)
[2022-07-25 23:36:27,174: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2dc410ab-b90e-47bb-bef8-4b747c407432, queue=awx_private_queue)
2022-07-25 23:36:27,191 DEBUG awx.main.dispatch delivered 2dc410ab-b90e-47bb-bef8-4b747c407432 to worker[183] qsize 0
2022-07-25 23:36:27,194 DEBUG awx.main.dispatch task 2dc410ab-b90e-47bb-bef8-4b747c407432 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:36:27,196 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:36:27,338 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:36:27,606 DEBUG awx.main.dispatch task 2dc410ab-b90e-47bb-bef8-4b747c407432 is finished
2022-07-25 23:36:46,399 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(7bc41766-629a-4839-a809-1b134b9cf3ad, queue=onap-awx-0)
[2022-07-25 23:36:46,399: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(7bc41766-629a-4839-a809-1b134b9cf3ad, queue=onap-awx-0)
2022-07-25 23:36:46,417 DEBUG awx.main.dispatch delivered 7bc41766-629a-4839-a809-1b134b9cf3ad to worker[181] qsize 0
2022-07-25 23:36:46,421 DEBUG awx.main.dispatch task 7bc41766-629a-4839-a809-1b134b9cf3ad starting awx.main.tasks.awx_isolated_heartbeat(*[])
2022-07-25 23:36:46,425 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2022-07-25 23:36:46,466 DEBUG awx.main.dispatch task 7bc41766-629a-4839-a809-1b134b9cf3ad is finished
2022-07-25 23:36:46,530 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(156d7b4e-da9c-49d9-9318-0e4020d7aa9e, queue=onap-awx-0)
[2022-07-25 23:36:46,530: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(156d7b4e-da9c-49d9-9318-0e4020d7aa9e, queue=onap-awx-0)
2022-07-25 23:36:46,583 DEBUG awx.main.dispatch delivered 156d7b4e-da9c-49d9-9318-0e4020d7aa9e to worker[182] qsize 0
2022-07-25 23:36:46,586 DEBUG awx.main.dispatch task 156d7b4e-da9c-49d9-9318-0e4020d7aa9e starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:36:46,587 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:36:46,657 DEBUG awx.main.dispatch task 156d7b4e-da9c-49d9-9318-0e4020d7aa9e is finished
2022-07-25 23:36:46,771 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2db1f74d-4d5b-4728-af3b-a057ada83873, queue=onap-awx-0)
[2022-07-25 23:36:46,771: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2db1f74d-4d5b-4728-af3b-a057ada83873, queue=onap-awx-0)
2022-07-25 23:36:46,791 DEBUG awx.main.dispatch delivered 2db1f74d-4d5b-4728-af3b-a057ada83873 to worker[180] qsize 0
2022-07-25 23:36:46,794 DEBUG awx.main.dispatch task 2db1f74d-4d5b-4728-af3b-a057ada83873 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:36:47,089 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(de55f04b-cdea-4b20-b2ac-65695f24ba64, queue=awx_private_queue)
[2022-07-25 23:36:47,089: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(de55f04b-cdea-4b20-b2ac-65695f24ba64, queue=awx_private_queue)
2022-07-25 23:36:47,106 DEBUG awx.main.dispatch delivered de55f04b-cdea-4b20-b2ac-65695f24ba64 to worker[181] qsize 0
2022-07-25 23:36:47,109 DEBUG awx.main.dispatch task de55f04b-cdea-4b20-b2ac-65695f24ba64 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:36:47,154 DEBUG awx.main.dispatch task 2db1f74d-4d5b-4728-af3b-a057ada83873 is finished
2022-07-25 23:36:47,174 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(54316826-b19e-44d7-adf5-f44f1a23da0e, queue=awx_private_queue)
[2022-07-25 23:36:47,174: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(54316826-b19e-44d7-adf5-f44f1a23da0e, queue=awx_private_queue)
2022-07-25 23:36:47,208 DEBUG awx.main.dispatch delivered 54316826-b19e-44d7-adf5-f44f1a23da0e to worker[183] qsize 0
2022-07-25 23:36:47,211 DEBUG awx.main.dispatch task 54316826-b19e-44d7-adf5-f44f1a23da0e starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:36:47,213 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:36:47,407 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:36:47,532 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:36:17.455959+00:00
2022-07-25 23:36:47,791 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:36:48,051 DEBUG awx.main.dispatch task de55f04b-cdea-4b20-b2ac-65695f24ba64 is finished
2022-07-25 23:36:48,599 DEBUG awx.main.dispatch task 54316826-b19e-44d7-adf5-f44f1a23da0e is finished
RESULT 2
OKREADY
2022-07-25 23:37:07,182 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fcc3d22d-955c-494d-b13b-9e6e9ceebf8b, queue=awx_private_queue)
[2022-07-25 23:37:07,182: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fcc3d22d-955c-494d-b13b-9e6e9ceebf8b, queue=awx_private_queue)
2022-07-25 23:37:07,200 DEBUG awx.main.dispatch delivered fcc3d22d-955c-494d-b13b-9e6e9ceebf8b to worker[181] qsize 0
2022-07-25 23:37:07,203 DEBUG awx.main.dispatch task fcc3d22d-955c-494d-b13b-9e6e9ceebf8b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:37:07,204 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:37:07,366 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:37:07,667 DEBUG awx.main.dispatch task fcc3d22d-955c-494d-b13b-9e6e9ceebf8b is finished
2022-07-25 23:37:17,089 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3512a550-6902-48bf-80b9-e8ca1b2b0d4d, queue=awx_private_queue)
[2022-07-25 23:37:17,089: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3512a550-6902-48bf-80b9-e8ca1b2b0d4d, queue=awx_private_queue)
2022-07-25 23:37:17,178 DEBUG awx.main.dispatch delivered 3512a550-6902-48bf-80b9-e8ca1b2b0d4d to worker[183] qsize 0
2022-07-25 23:37:17,181 DEBUG awx.main.dispatch task 3512a550-6902-48bf-80b9-e8ca1b2b0d4d starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:37:17,271 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:37:17,296 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:36:47.408031+00:00
2022-07-25 23:37:17,427 DEBUG awx.main.dispatch task 3512a550-6902-48bf-80b9-e8ca1b2b0d4d is finished
2022-07-25 23:37:27,183 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b4a5d0a7-ce9b-49e6-9494-06be0ce075fa, queue=awx_private_queue)
[2022-07-25 23:37:27,183: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b4a5d0a7-ce9b-49e6-9494-06be0ce075fa, queue=awx_private_queue)
2022-07-25 23:37:27,204 DEBUG awx.main.dispatch delivered b4a5d0a7-ce9b-49e6-9494-06be0ce075fa to worker[182] qsize 0
2022-07-25 23:37:27,207 DEBUG awx.main.dispatch task b4a5d0a7-ce9b-49e6-9494-06be0ce075fa starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:37:27,637 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:37:27,680 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:37:27,925 DEBUG awx.main.dispatch task b4a5d0a7-ce9b-49e6-9494-06be0ce075fa is finished
2022-07-25 23:37:46,541 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c9d912fe-f986-472d-9855-28c5dd9caa0e, queue=onap-awx-0)
[2022-07-25 23:37:46,541: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c9d912fe-f986-472d-9855-28c5dd9caa0e, queue=onap-awx-0)
2022-07-25 23:37:46,610 DEBUG awx.main.dispatch delivered c9d912fe-f986-472d-9855-28c5dd9caa0e to worker[180] qsize 0
2022-07-25 23:37:46,614 DEBUG awx.main.dispatch task c9d912fe-f986-472d-9855-28c5dd9caa0e starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:37:46,616 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:37:46,739 DEBUG awx.main.dispatch task c9d912fe-f986-472d-9855-28c5dd9caa0e is finished
2022-07-25 23:37:46,772 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(36cbcd5b-eba9-40b4-8b7c-ce2a70e0c350, queue=onap-awx-0)
[2022-07-25 23:37:46,772: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(36cbcd5b-eba9-40b4-8b7c-ce2a70e0c350, queue=onap-awx-0)
2022-07-25 23:37:46,790 DEBUG awx.main.dispatch delivered 36cbcd5b-eba9-40b4-8b7c-ce2a70e0c350 to worker[183] qsize 0
2022-07-25 23:37:46,793 DEBUG awx.main.dispatch task 36cbcd5b-eba9-40b4-8b7c-ce2a70e0c350 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:37:46,900 DEBUG awx.main.dispatch task 36cbcd5b-eba9-40b4-8b7c-ce2a70e0c350 is finished
2022-07-25 23:37:47,089 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(03d147be-8283-4d16-9271-5e3de998adb2, queue=awx_private_queue)
[2022-07-25 23:37:47,089: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(03d147be-8283-4d16-9271-5e3de998adb2, queue=awx_private_queue)
2022-07-25 23:37:47,124 DEBUG awx.main.dispatch delivered 03d147be-8283-4d16-9271-5e3de998adb2 to worker[181] qsize 0
2022-07-25 23:37:47,127 DEBUG awx.main.dispatch task 03d147be-8283-4d16-9271-5e3de998adb2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:37:47,183 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7c4eb15d-664d-4549-a9c2-c189fb077590, queue=awx_private_queue)
[2022-07-25 23:37:47,183: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7c4eb15d-664d-4549-a9c2-c189fb077590, queue=awx_private_queue)
2022-07-25 23:37:47,203 DEBUG awx.main.dispatch delivered 7c4eb15d-664d-4549-a9c2-c189fb077590 to worker[182] qsize 0
2022-07-25 23:37:47,207 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:37:47,207 DEBUG awx.main.dispatch task 7c4eb15d-664d-4549-a9c2-c189fb077590 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:37:47,208 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:37:47,235 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:37:17.271704+00:00
2022-07-25 23:37:47,304 DEBUG awx.main.dispatch task 03d147be-8283-4d16-9271-5e3de998adb2 is finished
2022-07-25 23:37:47,350 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:37:47,615 DEBUG awx.main.dispatch task 7c4eb15d-664d-4549-a9c2-c189fb077590 is finished
RESULT 2
OKREADY
2022-07-25 23:38:07,194 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b0c6ce56-a302-4e33-a784-b9b54584ea7b, queue=awx_private_queue)
[2022-07-25 23:38:07,194: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b0c6ce56-a302-4e33-a784-b9b54584ea7b, queue=awx_private_queue)
2022-07-25 23:38:07,213 DEBUG awx.main.dispatch delivered b0c6ce56-a302-4e33-a784-b9b54584ea7b to worker[180] qsize 0
2022-07-25 23:38:07,218 DEBUG awx.main.dispatch task b0c6ce56-a302-4e33-a784-b9b54584ea7b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:38:07,220 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:38:07,332 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:38:07,582 DEBUG awx.main.dispatch task b0c6ce56-a302-4e33-a784-b9b54584ea7b is finished
2022-07-25 23:38:17,091 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2fae918f-8fc4-49e2-921c-f3bca6cd05c6, queue=awx_private_queue)
[2022-07-25 23:38:17,091: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2fae918f-8fc4-49e2-921c-f3bca6cd05c6, queue=awx_private_queue)
2022-07-25 23:38:17,120 DEBUG awx.main.dispatch delivered 2fae918f-8fc4-49e2-921c-f3bca6cd05c6 to worker[180] qsize 0
2022-07-25 23:38:17,124 DEBUG awx.main.dispatch task 2fae918f-8fc4-49e2-921c-f3bca6cd05c6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:38:17,549 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:38:17,591 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:37:47.207696+00:00
2022-07-25 23:38:17,811 DEBUG awx.main.dispatch task 2fae918f-8fc4-49e2-921c-f3bca6cd05c6 is finished
2022-07-25 23:38:27,195 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(dcdf9579-f7eb-4690-a40c-60cbe0c94517, queue=awx_private_queue)
[2022-07-25 23:38:27,195: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(dcdf9579-f7eb-4690-a40c-60cbe0c94517, queue=awx_private_queue)
2022-07-25 23:38:27,319 DEBUG awx.main.dispatch delivered dcdf9579-f7eb-4690-a40c-60cbe0c94517 to worker[183] qsize 0
2022-07-25 23:38:27,322 DEBUG awx.main.dispatch task dcdf9579-f7eb-4690-a40c-60cbe0c94517 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:38:27,324 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:38:27,410 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:38:27,572 DEBUG awx.main.dispatch task dcdf9579-f7eb-4690-a40c-60cbe0c94517 is finished
2022-07-25 23:38:46,545 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(1fa1ef12-10b7-4c5e-bdfa-50ed666fc7b9, queue=onap-awx-0)
[2022-07-25 23:38:46,545: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(1fa1ef12-10b7-4c5e-bdfa-50ed666fc7b9, queue=onap-awx-0)
2022-07-25 23:38:46,612 DEBUG awx.main.dispatch delivered 1fa1ef12-10b7-4c5e-bdfa-50ed666fc7b9 to worker[180] qsize 0
2022-07-25 23:38:46,772 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(7fdabfdb-4ec0-428e-8334-e98f71833445, queue=onap-awx-0)
[2022-07-25 23:38:46,772: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(7fdabfdb-4ec0-428e-8334-e98f71833445, queue=onap-awx-0)
2022-07-25 23:38:46,789 DEBUG awx.main.dispatch delivered 7fdabfdb-4ec0-428e-8334-e98f71833445 to worker[182] qsize 0
2022-07-25 23:38:46,616 DEBUG awx.main.dispatch task 1fa1ef12-10b7-4c5e-bdfa-50ed666fc7b9 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:38:46,932 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:38:47,050 DEBUG awx.main.dispatch task 1fa1ef12-10b7-4c5e-bdfa-50ed666fc7b9 is finished
2022-07-25 23:38:47,091 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e83556dc-fbc8-4534-be32-aa49f22a6461, queue=awx_private_queue)
[2022-07-25 23:38:47,091: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e83556dc-fbc8-4534-be32-aa49f22a6461, queue=awx_private_queue)
2022-07-25 23:38:47,111 DEBUG awx.main.dispatch delivered e83556dc-fbc8-4534-be32-aa49f22a6461 to worker[180] qsize 0
2022-07-25 23:38:47,114 DEBUG awx.main.dispatch task e83556dc-fbc8-4534-be32-aa49f22a6461 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:38:46,792 DEBUG awx.main.dispatch task 7fdabfdb-4ec0-428e-8334-e98f71833445 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:38:47,194 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c8c85e84-051f-4411-b9cc-4b78599ed868, queue=awx_private_queue)
[2022-07-25 23:38:47,194: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c8c85e84-051f-4411-b9cc-4b78599ed868, queue=awx_private_queue)
2022-07-25 23:38:47,206 DEBUG awx.main.dispatch task 7fdabfdb-4ec0-428e-8334-e98f71833445 is finished
2022-07-25 23:38:47,214 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:38:47,213 DEBUG awx.main.dispatch delivered c8c85e84-051f-4411-b9cc-4b78599ed868 to worker[182] qsize 0
2022-07-25 23:38:47,216 DEBUG awx.main.dispatch task c8c85e84-051f-4411-b9cc-4b78599ed868 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:38:47,217 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:38:47,250 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:38:17.550230+00:00
2022-07-25 23:38:47,387 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:38:47,407 DEBUG awx.main.dispatch task e83556dc-fbc8-4534-be32-aa49f22a6461 is finished
2022-07-25 23:38:47,719 DEBUG awx.main.dispatch task c8c85e84-051f-4411-b9cc-4b78599ed868 is finished
RESULT 2
OKREADY
2022-07-25 23:39:07,199 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(be2530cb-5273-484d-9907-4db2d4e6c5f1, queue=awx_private_queue)
[2022-07-25 23:39:07,199: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(be2530cb-5273-484d-9907-4db2d4e6c5f1, queue=awx_private_queue)
2022-07-25 23:39:07,219 DEBUG awx.main.dispatch delivered be2530cb-5273-484d-9907-4db2d4e6c5f1 to worker[180] qsize 0
2022-07-25 23:39:07,223 DEBUG awx.main.dispatch task be2530cb-5273-484d-9907-4db2d4e6c5f1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:39:07,225 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:39:07,302 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:39:07,439 DEBUG awx.main.dispatch task be2530cb-5273-484d-9907-4db2d4e6c5f1 is finished
2022-07-25 23:39:17,089 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(81a1f810-abda-480b-8c05-14df933cbac9, queue=awx_private_queue)
[2022-07-25 23:39:17,089: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(81a1f810-abda-480b-8c05-14df933cbac9, queue=awx_private_queue)
2022-07-25 23:39:17,106 DEBUG awx.main.dispatch delivered 81a1f810-abda-480b-8c05-14df933cbac9 to worker[183] qsize 0
2022-07-25 23:39:17,108 DEBUG awx.main.dispatch task 81a1f810-abda-480b-8c05-14df933cbac9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:39:17,157 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:39:17,163 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:38:47.214608+00:00
2022-07-25 23:39:17,267 DEBUG awx.main.dispatch task 81a1f810-abda-480b-8c05-14df933cbac9 is finished
2022-07-25 23:39:27,199 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(99c632c2-4bfc-4746-ad38-622d837c3615, queue=awx_private_queue)
[2022-07-25 23:39:27,199: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(99c632c2-4bfc-4746-ad38-622d837c3615, queue=awx_private_queue)
2022-07-25 23:39:27,220 DEBUG awx.main.dispatch delivered 99c632c2-4bfc-4746-ad38-622d837c3615 to worker[181] qsize 0
2022-07-25 23:39:27,224 DEBUG awx.main.dispatch task 99c632c2-4bfc-4746-ad38-622d837c3615 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:39:27,228 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:39:27,400 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:39:27,692 DEBUG awx.main.dispatch task 99c632c2-4bfc-4746-ad38-622d837c3615 is finished
2022-07-25 23:39:46,553 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b22effce-e639-4f03-88f5-d382faba9fcc, queue=onap-awx-0)
[2022-07-25 23:39:46,553: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b22effce-e639-4f03-88f5-d382faba9fcc, queue=onap-awx-0)
2022-07-25 23:39:46,659 DEBUG awx.main.dispatch delivered b22effce-e639-4f03-88f5-d382faba9fcc to worker[180] qsize 0
2022-07-25 23:39:46,772 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(93d8cde1-6236-456c-a9f6-1750e1d460fc, queue=onap-awx-0)
[2022-07-25 23:39:46,772: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(93d8cde1-6236-456c-a9f6-1750e1d460fc, queue=onap-awx-0)
2022-07-25 23:39:46,789 DEBUG awx.main.dispatch delivered 93d8cde1-6236-456c-a9f6-1750e1d460fc to worker[181] qsize 0
2022-07-25 23:39:46,792 DEBUG awx.main.dispatch task 93d8cde1-6236-456c-a9f6-1750e1d460fc starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:39:46,907 DEBUG awx.main.dispatch task 93d8cde1-6236-456c-a9f6-1750e1d460fc is finished
2022-07-25 23:39:46,705 DEBUG awx.main.dispatch task b22effce-e639-4f03-88f5-d382faba9fcc starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:39:47,009 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:39:47,090 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(57f1f0ea-4006-407d-9039-f6c67f3ae0dd, queue=awx_private_queue)
[2022-07-25 23:39:47,090: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(57f1f0ea-4006-407d-9039-f6c67f3ae0dd, queue=awx_private_queue)
2022-07-25 23:39:47,113 DEBUG awx.main.dispatch delivered 57f1f0ea-4006-407d-9039-f6c67f3ae0dd to worker[181] qsize 0
2022-07-25 23:39:47,116 DEBUG awx.main.dispatch task 57f1f0ea-4006-407d-9039-f6c67f3ae0dd starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:39:47,141 DEBUG awx.main.dispatch task b22effce-e639-4f03-88f5-d382faba9fcc is finished
2022-07-25 23:39:47,195 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:39:47,200 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b92b86e9-1d69-42af-9dd3-4a4ef7ffabb0, queue=awx_private_queue)
[2022-07-25 23:39:47,200: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b92b86e9-1d69-42af-9dd3-4a4ef7ffabb0, queue=awx_private_queue)
2022-07-25 23:39:47,213 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:39:17.157324+00:00
2022-07-25 23:39:47,220 DEBUG awx.main.dispatch delivered b92b86e9-1d69-42af-9dd3-4a4ef7ffabb0 to worker[183] qsize 0
2022-07-25 23:39:47,223 DEBUG awx.main.dispatch task b92b86e9-1d69-42af-9dd3-4a4ef7ffabb0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:39:47,225 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:39:47,302 DEBUG awx.main.dispatch task 57f1f0ea-4006-407d-9039-f6c67f3ae0dd is finished
2022-07-25 23:39:47,370 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:39:47,734 DEBUG awx.main.dispatch task b92b86e9-1d69-42af-9dd3-4a4ef7ffabb0 is finished
RESULT 2
OKREADY
2022-07-25 23:40:07,207 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(64094013-d884-4399-b154-d9e68cfe1091, queue=awx_private_queue)
[2022-07-25 23:40:07,207: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(64094013-d884-4399-b154-d9e68cfe1091, queue=awx_private_queue)
2022-07-25 23:40:07,225 DEBUG awx.main.dispatch delivered 64094013-d884-4399-b154-d9e68cfe1091 to worker[181] qsize 0
2022-07-25 23:40:07,230 DEBUG awx.main.dispatch task 64094013-d884-4399-b154-d9e68cfe1091 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:40:07,232 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:40:07,333 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:40:07,429 DEBUG awx.main.dispatch task 64094013-d884-4399-b154-d9e68cfe1091 is finished
2022-07-25 23:40:17,090 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(658d5ef9-940b-49c6-87fc-674876407385, queue=awx_private_queue)
[2022-07-25 23:40:17,090: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(658d5ef9-940b-49c6-87fc-674876407385, queue=awx_private_queue)
2022-07-25 23:40:17,107 DEBUG awx.main.dispatch delivered 658d5ef9-940b-49c6-87fc-674876407385 to worker[181] qsize 0
2022-07-25 23:40:17,111 DEBUG awx.main.dispatch task 658d5ef9-940b-49c6-87fc-674876407385 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:40:17,157 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:40:17,173 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:39:47.196280+00:00
2022-07-25 23:40:17,256 DEBUG awx.main.dispatch task 658d5ef9-940b-49c6-87fc-674876407385 is finished
2022-07-25 23:40:27,207 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6f8445ac-144d-4ef2-9a63-be8b195b6d94, queue=awx_private_queue)
[2022-07-25 23:40:27,207: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6f8445ac-144d-4ef2-9a63-be8b195b6d94, queue=awx_private_queue)
2022-07-25 23:40:27,226 DEBUG awx.main.dispatch delivered 6f8445ac-144d-4ef2-9a63-be8b195b6d94 to worker[182] qsize 0
2022-07-25 23:40:27,229 DEBUG awx.main.dispatch task 6f8445ac-144d-4ef2-9a63-be8b195b6d94 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:40:27,232 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:40:27,377 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:40:27,617 DEBUG awx.main.dispatch task 6f8445ac-144d-4ef2-9a63-be8b195b6d94 is finished
2022-07-25 23:40:46,554 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(759b42b2-ae41-40fe-8c57-27637dceb042, queue=onap-awx-0)
[2022-07-25 23:40:46,554: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(759b42b2-ae41-40fe-8c57-27637dceb042, queue=onap-awx-0)
2022-07-25 23:40:46,710 DEBUG awx.main.dispatch delivered 759b42b2-ae41-40fe-8c57-27637dceb042 to worker[180] qsize 0
2022-07-25 23:40:46,772 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(14f8dafb-ed8b-4bda-bb4d-d0d2e50bb41d, queue=onap-awx-0)
[2022-07-25 23:40:46,772: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(14f8dafb-ed8b-4bda-bb4d-d0d2e50bb41d, queue=onap-awx-0)
2022-07-25 23:40:46,790 DEBUG awx.main.dispatch delivered 14f8dafb-ed8b-4bda-bb4d-d0d2e50bb41d to worker[181] qsize 0
2022-07-25 23:40:46,729 DEBUG awx.main.dispatch task 759b42b2-ae41-40fe-8c57-27637dceb042 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:40:47,083 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:40:46,793 DEBUG awx.main.dispatch task 14f8dafb-ed8b-4bda-bb4d-d0d2e50bb41d starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:40:47,089 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3da06275-1f2a-4b14-8a0f-348d78e0dc1f, queue=awx_private_queue)
[2022-07-25 23:40:47,089: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3da06275-1f2a-4b14-8a0f-348d78e0dc1f, queue=awx_private_queue)
2022-07-25 23:40:47,107 DEBUG awx.main.dispatch delivered 3da06275-1f2a-4b14-8a0f-348d78e0dc1f to worker[183] qsize 0
2022-07-25 23:40:47,110 DEBUG awx.main.dispatch task 3da06275-1f2a-4b14-8a0f-348d78e0dc1f starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:40:47,125 DEBUG awx.main.dispatch task 14f8dafb-ed8b-4bda-bb4d-d0d2e50bb41d is finished
2022-07-25 23:40:47,207 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3e96f8c2-29fe-457b-9884-b079a0cb8894, queue=awx_private_queue)
[2022-07-25 23:40:47,207: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3e96f8c2-29fe-457b-9884-b079a0cb8894, queue=awx_private_queue)
2022-07-25 23:40:47,225 DEBUG awx.main.dispatch delivered 3e96f8c2-29fe-457b-9884-b079a0cb8894 to worker[181] qsize 0
2022-07-25 23:40:47,229 DEBUG awx.main.dispatch task 3e96f8c2-29fe-457b-9884-b079a0cb8894 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:40:47,230 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:40:47,231 DEBUG awx.main.dispatch task 759b42b2-ae41-40fe-8c57-27637dceb042 is finished
2022-07-25 23:40:47,270 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:40:47,287 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:40:17.157843+00:00
2022-07-25 23:40:47,430 DEBUG awx.main.dispatch task 3da06275-1f2a-4b14-8a0f-348d78e0dc1f is finished
2022-07-25 23:40:47,681 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:40:48,491 DEBUG awx.main.dispatch task 3e96f8c2-29fe-457b-9884-b079a0cb8894 is finished
RESULT 2
OKREADY
2022-07-25 23:41:07,223 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4e15f02d-cdb0-4a20-8753-2668cf4d3238, queue=awx_private_queue)
[2022-07-25 23:41:07,223: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4e15f02d-cdb0-4a20-8753-2668cf4d3238, queue=awx_private_queue)
2022-07-25 23:41:07,243 DEBUG awx.main.dispatch delivered 4e15f02d-cdb0-4a20-8753-2668cf4d3238 to worker[183] qsize 0
2022-07-25 23:41:07,248 DEBUG awx.main.dispatch task 4e15f02d-cdb0-4a20-8753-2668cf4d3238 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:41:07,251 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:41:07,256 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:41:07,261 DEBUG awx.main.dispatch task 4e15f02d-cdb0-4a20-8753-2668cf4d3238 is finished
2022-07-25 23:41:17,091 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2c40ccb2-2a81-43b5-b4a6-32ed435d03db, queue=awx_private_queue)
[2022-07-25 23:41:17,091: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2c40ccb2-2a81-43b5-b4a6-32ed435d03db, queue=awx_private_queue)
2022-07-25 23:41:17,132 DEBUG awx.main.dispatch delivered 2c40ccb2-2a81-43b5-b4a6-32ed435d03db to worker[180] qsize 0
2022-07-25 23:41:17,135 DEBUG awx.main.dispatch task 2c40ccb2-2a81-43b5-b4a6-32ed435d03db starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:41:17,141 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:41:17,145 DEBUG awx.main.dispatch task 2c40ccb2-2a81-43b5-b4a6-32ed435d03db is finished
2022-07-25 23:41:27,223 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b15b34ae-4157-468a-963c-f5523afe32b1, queue=awx_private_queue)
[2022-07-25 23:41:27,223: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b15b34ae-4157-468a-963c-f5523afe32b1, queue=awx_private_queue)
2022-07-25 23:41:27,255 DEBUG awx.main.dispatch delivered b15b34ae-4157-468a-963c-f5523afe32b1 to worker[183] qsize 0
2022-07-25 23:41:27,258 DEBUG awx.main.dispatch task b15b34ae-4157-468a-963c-f5523afe32b1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:41:27,260 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:41:27,264 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:41:27,266 DEBUG awx.main.dispatch task b15b34ae-4157-468a-963c-f5523afe32b1 is finished
2022-07-25 23:41:46,565 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(3773a03d-51d3-48da-b45f-88e69a0ffab5, queue=onap-awx-0)
[2022-07-25 23:41:46,565: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(3773a03d-51d3-48da-b45f-88e69a0ffab5, queue=onap-awx-0)
2022-07-25 23:41:46,669 DEBUG awx.main.dispatch delivered 3773a03d-51d3-48da-b45f-88e69a0ffab5 to worker[182] qsize 0
2022-07-25 23:41:46,674 DEBUG awx.main.dispatch task 3773a03d-51d3-48da-b45f-88e69a0ffab5 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:41:46,678 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:41:46,685 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:41:46,689 DEBUG awx.main.dispatch task 3773a03d-51d3-48da-b45f-88e69a0ffab5 is finished
2022-07-25 23:41:46,772 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3c1d8c4c-402e-4f87-b329-b159e59cbbf3, queue=onap-awx-0)
[2022-07-25 23:41:46,772: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3c1d8c4c-402e-4f87-b329-b159e59cbbf3, queue=onap-awx-0)
2022-07-25 23:41:46,794 DEBUG awx.main.dispatch delivered 3c1d8c4c-402e-4f87-b329-b159e59cbbf3 to worker[181] qsize 0
2022-07-25 23:41:46,815 DEBUG awx.main.dispatch task 3c1d8c4c-402e-4f87-b329-b159e59cbbf3 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:41:46,829 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:41:46,832 DEBUG awx.main.dispatch task 3c1d8c4c-402e-4f87-b329-b159e59cbbf3 is finished
2022-07-25 23:41:47,090 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6752fdb5-9d0c-4eab-a5e6-d52502697a8f, queue=awx_private_queue)
[2022-07-25 23:41:47,090: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6752fdb5-9d0c-4eab-a5e6-d52502697a8f, queue=awx_private_queue)
2022-07-25 23:41:47,110 DEBUG awx.main.dispatch delivered 6752fdb5-9d0c-4eab-a5e6-d52502697a8f to worker[182] qsize 0
2022-07-25 23:41:47,114 DEBUG awx.main.dispatch task 6752fdb5-9d0c-4eab-a5e6-d52502697a8f starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:41:47,117 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:41:47,120 DEBUG awx.main.dispatch task 6752fdb5-9d0c-4eab-a5e6-d52502697a8f is finished
2022-07-25 23:41:47,224 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5405c477-6e22-4c0f-a0e7-c1dfe5d9595f, queue=awx_private_queue)
[2022-07-25 23:41:47,224: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5405c477-6e22-4c0f-a0e7-c1dfe5d9595f, queue=awx_private_queue)
2022-07-25 23:41:47,254 DEBUG awx.main.dispatch delivered 5405c477-6e22-4c0f-a0e7-c1dfe5d9595f to worker[183] qsize 0
2022-07-25 23:41:47,257 DEBUG awx.main.dispatch task 5405c477-6e22-4c0f-a0e7-c1dfe5d9595f starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:41:47,270 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:41:47,272 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:41:47,274 DEBUG awx.main.dispatch task 5405c477-6e22-4c0f-a0e7-c1dfe5d9595f is finished
RESULT 2
OKREADY
2022-07-25 23:42:07,231 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bffdd649-2307-4d31-a5e1-857861508422, queue=awx_private_queue)
[2022-07-25 23:42:07,231: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bffdd649-2307-4d31-a5e1-857861508422, queue=awx_private_queue)
2022-07-25 23:42:07,250 DEBUG awx.main.dispatch delivered bffdd649-2307-4d31-a5e1-857861508422 to worker[180] qsize 0
2022-07-25 23:42:07,454 DEBUG awx.main.dispatch task bffdd649-2307-4d31-a5e1-857861508422 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:42:07,457 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:42:07,531 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:42:07,639 DEBUG awx.main.dispatch task bffdd649-2307-4d31-a5e1-857861508422 is finished
2022-07-25 23:42:17,090 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1957ca71-a542-41e6-b2d2-d0bccff4bac5, queue=awx_private_queue)
[2022-07-25 23:42:17,090: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1957ca71-a542-41e6-b2d2-d0bccff4bac5, queue=awx_private_queue)
2022-07-25 23:42:17,108 DEBUG awx.main.dispatch delivered 1957ca71-a542-41e6-b2d2-d0bccff4bac5 to worker[183] qsize 0
2022-07-25 23:42:17,111 DEBUG awx.main.dispatch task 1957ca71-a542-41e6-b2d2-d0bccff4bac5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:42:17,196 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:42:17,223 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:40:47.271109+00:00
2022-07-25 23:42:17,329 DEBUG awx.main.dispatch task 1957ca71-a542-41e6-b2d2-d0bccff4bac5 is finished
2022-07-25 23:42:27,231 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(456791f5-f24a-4feb-ac49-844e7775d488, queue=awx_private_queue)
[2022-07-25 23:42:27,231: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(456791f5-f24a-4feb-ac49-844e7775d488, queue=awx_private_queue)
2022-07-25 23:42:27,259 DEBUG awx.main.dispatch delivered 456791f5-f24a-4feb-ac49-844e7775d488 to worker[182] qsize 0
2022-07-25 23:42:27,262 DEBUG awx.main.dispatch task 456791f5-f24a-4feb-ac49-844e7775d488 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:42:27,264 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:42:27,381 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:42:27,693 DEBUG awx.main.dispatch task 456791f5-f24a-4feb-ac49-844e7775d488 is finished
2022-07-25 23:42:46,571 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(55bc3e20-e953-47b3-b780-5520e53a3042, queue=onap-awx-0)
[2022-07-25 23:42:46,571: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(55bc3e20-e953-47b3-b780-5520e53a3042, queue=onap-awx-0)
2022-07-25 23:42:46,687 DEBUG awx.main.dispatch delivered 55bc3e20-e953-47b3-b780-5520e53a3042 to worker[183] qsize 0
2022-07-25 23:42:46,689 DEBUG awx.main.dispatch task 55bc3e20-e953-47b3-b780-5520e53a3042 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:42:46,691 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:42:46,772 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5099b11e-8119-4662-9172-9df0fa6745d6, queue=onap-awx-0)
[2022-07-25 23:42:46,772: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5099b11e-8119-4662-9172-9df0fa6745d6, queue=onap-awx-0)
2022-07-25 23:42:46,797 DEBUG awx.main.dispatch delivered 5099b11e-8119-4662-9172-9df0fa6745d6 to worker[182] qsize 0
2022-07-25 23:42:46,799 DEBUG awx.main.dispatch task 5099b11e-8119-4662-9172-9df0fa6745d6 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:42:46,940 DEBUG awx.main.dispatch task 5099b11e-8119-4662-9172-9df0fa6745d6 is finished
2022-07-25 23:42:47,090 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(989391b5-1a26-4cea-826d-01a3f14bfb71, queue=awx_private_queue)
[2022-07-25 23:42:47,090: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(989391b5-1a26-4cea-826d-01a3f14bfb71, queue=awx_private_queue)
2022-07-25 23:42:47,110 DEBUG awx.main.dispatch delivered 989391b5-1a26-4cea-826d-01a3f14bfb71 to worker[180] qsize 0
2022-07-25 23:42:47,114 DEBUG awx.main.dispatch task 989391b5-1a26-4cea-826d-01a3f14bfb71 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:42:47,164 DEBUG awx.main.dispatch task 55bc3e20-e953-47b3-b780-5520e53a3042 is finished
2022-07-25 23:42:47,179 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:42:47,194 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:42:17.197017+00:00
2022-07-25 23:42:47,231 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(58034850-b997-4599-ad3a-39e012f9b18b, queue=awx_private_queue)
[2022-07-25 23:42:47,231: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(58034850-b997-4599-ad3a-39e012f9b18b, queue=awx_private_queue)
2022-07-25 23:42:47,263 DEBUG awx.main.dispatch delivered 58034850-b997-4599-ad3a-39e012f9b18b to worker[181] qsize 0
2022-07-25 23:42:47,265 DEBUG awx.main.dispatch task 58034850-b997-4599-ad3a-39e012f9b18b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:42:47,268 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:42:47,332 DEBUG awx.main.dispatch task 989391b5-1a26-4cea-826d-01a3f14bfb71 is finished
2022-07-25 23:42:47,433 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:42:47,545 DEBUG awx.main.dispatch task 58034850-b997-4599-ad3a-39e012f9b18b is finished
RESULT 2
OKREADY
2022-07-25 23:43:07,242 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3de9ba77-0985-457e-89e5-9ae4e3849c14, queue=awx_private_queue)
[2022-07-25 23:43:07,242: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3de9ba77-0985-457e-89e5-9ae4e3849c14, queue=awx_private_queue)
2022-07-25 23:43:07,379 DEBUG awx.main.dispatch delivered 3de9ba77-0985-457e-89e5-9ae4e3849c14 to worker[183] qsize 0
2022-07-25 23:43:07,611 DEBUG awx.main.dispatch task 3de9ba77-0985-457e-89e5-9ae4e3849c14 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:43:07,615 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:43:07,672 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:43:07,733 DEBUG awx.main.dispatch task 3de9ba77-0985-457e-89e5-9ae4e3849c14 is finished
2022-07-25 23:43:17,091 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c0fcd219-9e95-4a3d-b7be-d2ef8b855c2d, queue=awx_private_queue)
[2022-07-25 23:43:17,091: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c0fcd219-9e95-4a3d-b7be-d2ef8b855c2d, queue=awx_private_queue)
2022-07-25 23:43:17,119 DEBUG awx.main.dispatch delivered c0fcd219-9e95-4a3d-b7be-d2ef8b855c2d to worker[180] qsize 0
2022-07-25 23:43:17,123 DEBUG awx.main.dispatch task c0fcd219-9e95-4a3d-b7be-d2ef8b855c2d starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:43:17,279 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:43:17,312 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:42:47.179940+00:00
2022-07-25 23:43:17,504 DEBUG awx.main.dispatch task c0fcd219-9e95-4a3d-b7be-d2ef8b855c2d is finished
2022-07-25 23:43:27,242 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(71348b6b-5290-4b8c-99f6-b847a5b69cfb, queue=awx_private_queue)
[2022-07-25 23:43:27,242: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(71348b6b-5290-4b8c-99f6-b847a5b69cfb, queue=awx_private_queue)
2022-07-25 23:43:27,268 DEBUG awx.main.dispatch delivered 71348b6b-5290-4b8c-99f6-b847a5b69cfb to worker[181] qsize 0
2022-07-25 23:43:27,271 DEBUG awx.main.dispatch task 71348b6b-5290-4b8c-99f6-b847a5b69cfb starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:43:27,272 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:43:27,339 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:43:27,519 DEBUG awx.main.dispatch task 71348b6b-5290-4b8c-99f6-b847a5b69cfb is finished
2022-07-25 23:43:46,580 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(9b5a1f59-4ac1-40c9-89bf-b3a58a8c1e2e, queue=onap-awx-0)
[2022-07-25 23:43:46,580: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(9b5a1f59-4ac1-40c9-89bf-b3a58a8c1e2e, queue=onap-awx-0)
2022-07-25 23:43:46,730 DEBUG awx.main.dispatch delivered 9b5a1f59-4ac1-40c9-89bf-b3a58a8c1e2e to worker[183] qsize 0
2022-07-25 23:43:46,733 DEBUG awx.main.dispatch task 9b5a1f59-4ac1-40c9-89bf-b3a58a8c1e2e starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:43:46,735 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:43:46,772 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(4f78c7ac-dcd7-4600-b889-2605987c2eaf, queue=onap-awx-0)
[2022-07-25 23:43:46,772: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(4f78c7ac-dcd7-4600-b889-2605987c2eaf, queue=onap-awx-0)
2022-07-25 23:43:46,789 DEBUG awx.main.dispatch delivered 4f78c7ac-dcd7-4600-b889-2605987c2eaf to worker[180] qsize 0
2022-07-25 23:43:46,792 DEBUG awx.main.dispatch task 4f78c7ac-dcd7-4600-b889-2605987c2eaf starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:43:46,922 DEBUG awx.main.dispatch task 9b5a1f59-4ac1-40c9-89bf-b3a58a8c1e2e is finished
2022-07-25 23:43:46,925 DEBUG awx.main.dispatch task 4f78c7ac-dcd7-4600-b889-2605987c2eaf is finished
2022-07-25 23:43:47,090 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4485fec7-acb5-4745-9ea3-c49767487e6d, queue=awx_private_queue)
[2022-07-25 23:43:47,090: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4485fec7-acb5-4745-9ea3-c49767487e6d, queue=awx_private_queue)
2022-07-25 23:43:47,107 DEBUG awx.main.dispatch delivered 4485fec7-acb5-4745-9ea3-c49767487e6d to worker[180] qsize 0
2022-07-25 23:43:47,109 DEBUG awx.main.dispatch task 4485fec7-acb5-4745-9ea3-c49767487e6d starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:43:47,242 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(02e4311b-b8e6-4928-8f9a-1586b7d313e7, queue=awx_private_queue)
[2022-07-25 23:43:47,242: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(02e4311b-b8e6-4928-8f9a-1586b7d313e7, queue=awx_private_queue)
2022-07-25 23:43:47,258 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:43:47,260 DEBUG awx.main.dispatch delivered 02e4311b-b8e6-4928-8f9a-1586b7d313e7 to worker[181] qsize 0
2022-07-25 23:43:47,262 DEBUG awx.main.dispatch task 02e4311b-b8e6-4928-8f9a-1586b7d313e7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:43:47,264 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:43:47,277 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:43:17.280219+00:00
2022-07-25 23:43:47,344 DEBUG awx.main.dispatch task 4485fec7-acb5-4745-9ea3-c49767487e6d is finished
2022-07-25 23:43:47,387 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:43:47,543 DEBUG awx.main.dispatch task 02e4311b-b8e6-4928-8f9a-1586b7d313e7 is finished
RESULT 2
OKREADY
2022-07-25 23:44:07,256 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8f001f21-21bb-4d86-8619-f9c14b0912b5, queue=awx_private_queue)
[2022-07-25 23:44:07,256: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8f001f21-21bb-4d86-8619-f9c14b0912b5, queue=awx_private_queue)
2022-07-25 23:44:07,373 DEBUG awx.main.dispatch delivered 8f001f21-21bb-4d86-8619-f9c14b0912b5 to worker[182] qsize 0
2022-07-25 23:44:07,377 DEBUG awx.main.dispatch task 8f001f21-21bb-4d86-8619-f9c14b0912b5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:44:07,659 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:44:07,755 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:44:07,954 DEBUG awx.main.dispatch task 8f001f21-21bb-4d86-8619-f9c14b0912b5 is finished
2022-07-25 23:44:17,091 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(26883e00-aff2-42af-875a-46b03b7b47a1, queue=awx_private_queue)
[2022-07-25 23:44:17,091: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(26883e00-aff2-42af-875a-46b03b7b47a1, queue=awx_private_queue)
2022-07-25 23:44:17,110 DEBUG awx.main.dispatch delivered 26883e00-aff2-42af-875a-46b03b7b47a1 to worker[180] qsize 0
2022-07-25 23:44:17,114 DEBUG awx.main.dispatch task 26883e00-aff2-42af-875a-46b03b7b47a1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:44:17,173 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:44:17,201 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:43:47.258328+00:00
2022-07-25 23:44:17,383 DEBUG awx.main.dispatch task 26883e00-aff2-42af-875a-46b03b7b47a1 is finished
2022-07-25 23:44:27,255 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ef678fae-b76d-40ba-8a75-e9c667dc7647, queue=awx_private_queue)
[2022-07-25 23:44:27,255: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ef678fae-b76d-40ba-8a75-e9c667dc7647, queue=awx_private_queue)
2022-07-25 23:44:27,274 DEBUG awx.main.dispatch delivered ef678fae-b76d-40ba-8a75-e9c667dc7647 to worker[183] qsize 0
2022-07-25 23:44:27,277 DEBUG awx.main.dispatch task ef678fae-b76d-40ba-8a75-e9c667dc7647 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:44:27,280 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:44:27,361 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:44:27,482 DEBUG awx.main.dispatch task ef678fae-b76d-40ba-8a75-e9c667dc7647 is finished
2022-07-25 23:44:46,586 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(55f59be1-8e6d-44f2-84fb-1d9885a96c17, queue=onap-awx-0)
[2022-07-25 23:44:46,586: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(55f59be1-8e6d-44f2-84fb-1d9885a96c17, queue=onap-awx-0)
2022-07-25 23:44:46,646 DEBUG awx.main.dispatch delivered 55f59be1-8e6d-44f2-84fb-1d9885a96c17 to worker[182] qsize 0
2022-07-25 23:44:46,649 DEBUG awx.main.dispatch task 55f59be1-8e6d-44f2-84fb-1d9885a96c17 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:44:46,651 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:44:46,699 DEBUG awx.main.dispatch task 55f59be1-8e6d-44f2-84fb-1d9885a96c17 is finished
2022-07-25 23:44:46,772 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(dcbc2d48-eeef-405e-aa05-241ed42ceda2, queue=onap-awx-0)
[2022-07-25 23:44:46,772: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(dcbc2d48-eeef-405e-aa05-241ed42ceda2, queue=onap-awx-0)
2022-07-25 23:44:46,790 DEBUG awx.main.dispatch delivered dcbc2d48-eeef-405e-aa05-241ed42ceda2 to worker[180] qsize 0
2022-07-25 23:44:46,793 DEBUG awx.main.dispatch task dcbc2d48-eeef-405e-aa05-241ed42ceda2 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:44:46,826 DEBUG awx.main.dispatch task dcbc2d48-eeef-405e-aa05-241ed42ceda2 is finished
2022-07-25 23:44:47,091 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9b017211-ce7a-47fb-b175-c9bbde968886, queue=awx_private_queue)
[2022-07-25 23:44:47,091: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9b017211-ce7a-47fb-b175-c9bbde968886, queue=awx_private_queue)
2022-07-25 23:44:47,108 DEBUG awx.main.dispatch delivered 9b017211-ce7a-47fb-b175-c9bbde968886 to worker[180] qsize 0
2022-07-25 23:44:47,111 DEBUG awx.main.dispatch task 9b017211-ce7a-47fb-b175-c9bbde968886 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:44:47,144 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:44:47,149 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:44:17.174220+00:00
2022-07-25 23:44:47,180 DEBUG awx.main.dispatch task 9b017211-ce7a-47fb-b175-c9bbde968886 is finished
2022-07-25 23:44:47,255 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(78b598bf-569c-4462-93b2-f4fedb1f5e3b, queue=awx_private_queue)
[2022-07-25 23:44:47,255: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(78b598bf-569c-4462-93b2-f4fedb1f5e3b, queue=awx_private_queue)
2022-07-25 23:44:47,275 DEBUG awx.main.dispatch delivered 78b598bf-569c-4462-93b2-f4fedb1f5e3b to worker[180] qsize 0
2022-07-25 23:44:47,278 DEBUG awx.main.dispatch task 78b598bf-569c-4462-93b2-f4fedb1f5e3b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:44:47,281 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:44:47,328 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:44:47,454 DEBUG awx.main.dispatch task 78b598bf-569c-4462-93b2-f4fedb1f5e3b is finished
RESULT 2
OKREADY
2022-07-25 23:45:07,258 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7bc928e1-011c-47a0-b8e6-ec47af2f8ec1, queue=awx_private_queue)
[2022-07-25 23:45:07,258: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7bc928e1-011c-47a0-b8e6-ec47af2f8ec1, queue=awx_private_queue)
2022-07-25 23:45:07,393 DEBUG awx.main.dispatch delivered 7bc928e1-011c-47a0-b8e6-ec47af2f8ec1 to worker[183] qsize 0
2022-07-25 23:45:07,396 DEBUG awx.main.dispatch task 7bc928e1-011c-47a0-b8e6-ec47af2f8ec1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:45:07,397 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:45:07,578 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:45:07,814 DEBUG awx.main.dispatch task 7bc928e1-011c-47a0-b8e6-ec47af2f8ec1 is finished
2022-07-25 23:45:17,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(aa5dc81b-e804-4518-b698-aeb4f110fc34, queue=awx_private_queue)
[2022-07-25 23:45:17,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(aa5dc81b-e804-4518-b698-aeb4f110fc34, queue=awx_private_queue)
2022-07-25 23:45:17,111 DEBUG awx.main.dispatch delivered aa5dc81b-e804-4518-b698-aeb4f110fc34 to worker[182] qsize 0
2022-07-25 23:45:17,344 DEBUG awx.main.dispatch task aa5dc81b-e804-4518-b698-aeb4f110fc34 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:45:17,487 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:45:17,547 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:44:47.144697+00:00
2022-07-25 23:45:17,807 DEBUG awx.main.dispatch task aa5dc81b-e804-4518-b698-aeb4f110fc34 is finished
2022-07-25 23:45:27,258 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(756c2b1f-0128-40a2-8394-65e702024d8f, queue=awx_private_queue)
[2022-07-25 23:45:27,258: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(756c2b1f-0128-40a2-8394-65e702024d8f, queue=awx_private_queue)
2022-07-25 23:45:27,277 DEBUG awx.main.dispatch delivered 756c2b1f-0128-40a2-8394-65e702024d8f to worker[183] qsize 0
2022-07-25 23:45:27,280 DEBUG awx.main.dispatch task 756c2b1f-0128-40a2-8394-65e702024d8f starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:45:27,282 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:45:28,043 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:45:28,490 DEBUG awx.main.dispatch task 756c2b1f-0128-40a2-8394-65e702024d8f is finished
2022-07-25 23:45:46,592 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(714ebd60-4102-4af1-b606-b9df0f5caa3c, queue=onap-awx-0)
[2022-07-25 23:45:46,592: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(714ebd60-4102-4af1-b606-b9df0f5caa3c, queue=onap-awx-0)
2022-07-25 23:45:46,773 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(857ad0e4-d168-4462-a8d0-5a75b933dc7f, queue=onap-awx-0)
[2022-07-25 23:45:46,773: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(857ad0e4-d168-4462-a8d0-5a75b933dc7f, queue=onap-awx-0)
2022-07-25 23:45:46,783 DEBUG awx.main.dispatch delivered 714ebd60-4102-4af1-b606-b9df0f5caa3c to worker[181] qsize 0
2022-07-25 23:45:46,786 DEBUG awx.main.dispatch task 714ebd60-4102-4af1-b606-b9df0f5caa3c starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:45:46,790 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:45:46,791 DEBUG awx.main.dispatch delivered 857ad0e4-d168-4462-a8d0-5a75b933dc7f to worker[180] qsize 0
2022-07-25 23:45:46,792 DEBUG awx.main.dispatch task 857ad0e4-d168-4462-a8d0-5a75b933dc7f starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:45:46,800 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:45:46,799 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:45:46,803 DEBUG awx.main.dispatch task 714ebd60-4102-4af1-b606-b9df0f5caa3c is finished
2022-07-25 23:45:46,803 DEBUG awx.main.dispatch task 857ad0e4-d168-4462-a8d0-5a75b933dc7f is finished
2022-07-25 23:45:47,091 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(791b1c17-1d89-4fe0-93bf-074c2af0dc7a, queue=awx_private_queue)
[2022-07-25 23:45:47,091: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(791b1c17-1d89-4fe0-93bf-074c2af0dc7a, queue=awx_private_queue)
2022-07-25 23:45:47,109 DEBUG awx.main.dispatch delivered 791b1c17-1d89-4fe0-93bf-074c2af0dc7a to worker[183] qsize 0
2022-07-25 23:45:47,112 DEBUG awx.main.dispatch task 791b1c17-1d89-4fe0-93bf-074c2af0dc7a starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:45:47,115 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:45:47,117 DEBUG awx.main.dispatch task 791b1c17-1d89-4fe0-93bf-074c2af0dc7a is finished
2022-07-25 23:45:47,259 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(edfbb510-5ed4-4d31-a821-2408dbc1210a, queue=awx_private_queue)
[2022-07-25 23:45:47,259: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(edfbb510-5ed4-4d31-a821-2408dbc1210a, queue=awx_private_queue)
2022-07-25 23:45:47,278 DEBUG awx.main.dispatch delivered edfbb510-5ed4-4d31-a821-2408dbc1210a to worker[183] qsize 0
2022-07-25 23:45:47,281 DEBUG awx.main.dispatch task edfbb510-5ed4-4d31-a821-2408dbc1210a starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:45:47,283 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:45:47,285 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:45:47,288 DEBUG awx.main.dispatch task edfbb510-5ed4-4d31-a821-2408dbc1210a is finished
RESULT 2
OKREADY
2022-07-25 23:46:07,269 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3e392d90-6b60-4f37-8c9c-bc79d8f416c6, queue=awx_private_queue)
[2022-07-25 23:46:07,269: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3e392d90-6b60-4f37-8c9c-bc79d8f416c6, queue=awx_private_queue)
2022-07-25 23:46:07,291 DEBUG awx.main.dispatch delivered 3e392d90-6b60-4f37-8c9c-bc79d8f416c6 to worker[182] qsize 0
2022-07-25 23:46:07,294 DEBUG awx.main.dispatch task 3e392d90-6b60-4f37-8c9c-bc79d8f416c6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:46:07,296 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:46:07,300 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:46:07,304 DEBUG awx.main.dispatch task 3e392d90-6b60-4f37-8c9c-bc79d8f416c6 is finished
2022-07-25 23:46:17,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4cdfa8a4-706f-4d4c-8488-34a75bfc4c88, queue=awx_private_queue)
[2022-07-25 23:46:17,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4cdfa8a4-706f-4d4c-8488-34a75bfc4c88, queue=awx_private_queue)
2022-07-25 23:46:17,115 DEBUG awx.main.dispatch delivered 4cdfa8a4-706f-4d4c-8488-34a75bfc4c88 to worker[183] qsize 0
2022-07-25 23:46:17,327 DEBUG awx.main.dispatch task 4cdfa8a4-706f-4d4c-8488-34a75bfc4c88 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:46:17,331 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:46:17,333 DEBUG awx.main.dispatch task 4cdfa8a4-706f-4d4c-8488-34a75bfc4c88 is finished
2022-07-25 23:46:27,269 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f8324731-e013-4044-9ce8-713966d90f3c, queue=awx_private_queue)
[2022-07-25 23:46:27,269: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f8324731-e013-4044-9ce8-713966d90f3c, queue=awx_private_queue)
2022-07-25 23:46:27,287 DEBUG awx.main.dispatch delivered f8324731-e013-4044-9ce8-713966d90f3c to worker[181] qsize 0
2022-07-25 23:46:27,289 DEBUG awx.main.dispatch task f8324731-e013-4044-9ce8-713966d90f3c starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:46:27,291 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:46:27,295 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:46:27,298 DEBUG awx.main.dispatch task f8324731-e013-4044-9ce8-713966d90f3c is finished
2022-07-25 23:46:46,408 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(50da9447-a867-49c0-91ec-a2e5aaa2cc29, queue=onap-awx-0)
[2022-07-25 23:46:46,408: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(50da9447-a867-49c0-91ec-a2e5aaa2cc29, queue=onap-awx-0)
2022-07-25 23:46:46,430 DEBUG awx.main.dispatch delivered 50da9447-a867-49c0-91ec-a2e5aaa2cc29 to worker[183] qsize 0
2022-07-25 23:46:46,434 DEBUG awx.main.dispatch task 50da9447-a867-49c0-91ec-a2e5aaa2cc29 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2022-07-25 23:46:46,437 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2022-07-25 23:46:46,546 DEBUG awx.main.dispatch task 50da9447-a867-49c0-91ec-a2e5aaa2cc29 is finished
2022-07-25 23:46:46,592 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5b8fa407-1210-49ca-961e-6af9e0c4a479, queue=onap-awx-0)
[2022-07-25 23:46:46,592: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5b8fa407-1210-49ca-961e-6af9e0c4a479, queue=onap-awx-0)
2022-07-25 23:46:46,663 DEBUG awx.main.dispatch delivered 5b8fa407-1210-49ca-961e-6af9e0c4a479 to worker[181] qsize 0
2022-07-25 23:46:46,667 DEBUG awx.main.dispatch task 5b8fa407-1210-49ca-961e-6af9e0c4a479 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:46:46,669 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:46:46,772 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(355d9aff-cc1b-4f95-b59a-77065fc39920, queue=onap-awx-0)
[2022-07-25 23:46:46,772: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(355d9aff-cc1b-4f95-b59a-77065fc39920, queue=onap-awx-0)
2022-07-25 23:46:46,792 DEBUG awx.main.dispatch delivered 355d9aff-cc1b-4f95-b59a-77065fc39920 to worker[180] qsize 0
2022-07-25 23:46:46,794 DEBUG awx.main.dispatch task 355d9aff-cc1b-4f95-b59a-77065fc39920 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:46:46,863 DEBUG awx.main.dispatch task 5b8fa407-1210-49ca-961e-6af9e0c4a479 is finished
2022-07-25 23:46:46,898 DEBUG awx.main.dispatch task 355d9aff-cc1b-4f95-b59a-77065fc39920 is finished
2022-07-25 23:46:47,091 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(97beadfb-6f52-41fe-bc94-76d14457986a, queue=awx_private_queue)
[2022-07-25 23:46:47,091: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(97beadfb-6f52-41fe-bc94-76d14457986a, queue=awx_private_queue)
2022-07-25 23:46:47,110 DEBUG awx.main.dispatch delivered 97beadfb-6f52-41fe-bc94-76d14457986a to worker[182] qsize 0
2022-07-25 23:46:47,113 DEBUG awx.main.dispatch task 97beadfb-6f52-41fe-bc94-76d14457986a starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:46:47,187 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:46:47,196 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:45:17.487475+00:00
2022-07-25 23:46:47,253 DEBUG awx.main.dispatch task 97beadfb-6f52-41fe-bc94-76d14457986a is finished
2022-07-25 23:46:47,269 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(caca72b7-7883-4851-a5ce-3dc71195dc73, queue=awx_private_queue)
[2022-07-25 23:46:47,269: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(caca72b7-7883-4851-a5ce-3dc71195dc73, queue=awx_private_queue)
2022-07-25 23:46:47,286 DEBUG awx.main.dispatch delivered caca72b7-7883-4851-a5ce-3dc71195dc73 to worker[181] qsize 0
2022-07-25 23:46:47,288 DEBUG awx.main.dispatch task caca72b7-7883-4851-a5ce-3dc71195dc73 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:46:47,290 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:46:47,470 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:46:47,598 DEBUG awx.main.dispatch task caca72b7-7883-4851-a5ce-3dc71195dc73 is finished
RESULT 2
OKREADY
2022-07-25 23:47:07,283 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(220af6db-f80e-4b7c-be58-12e99af2e333, queue=awx_private_queue)
[2022-07-25 23:47:07,283: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(220af6db-f80e-4b7c-be58-12e99af2e333, queue=awx_private_queue)
2022-07-25 23:47:07,303 DEBUG awx.main.dispatch delivered 220af6db-f80e-4b7c-be58-12e99af2e333 to worker[180] qsize 0
2022-07-25 23:47:07,307 DEBUG awx.main.dispatch task 220af6db-f80e-4b7c-be58-12e99af2e333 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:47:07,310 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:47:07,418 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:47:07,588 DEBUG awx.main.dispatch task 220af6db-f80e-4b7c-be58-12e99af2e333 is finished
2022-07-25 23:47:17,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5e088ad5-ca9e-4844-8a4e-8247bea8c15b, queue=awx_private_queue)
[2022-07-25 23:47:17,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5e088ad5-ca9e-4844-8a4e-8247bea8c15b, queue=awx_private_queue)
2022-07-25 23:47:17,159 DEBUG awx.main.dispatch delivered 5e088ad5-ca9e-4844-8a4e-8247bea8c15b to worker[182] qsize 0
2022-07-25 23:47:17,404 DEBUG awx.main.dispatch task 5e088ad5-ca9e-4844-8a4e-8247bea8c15b starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:47:17,444 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:47:17,450 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:46:47.187883+00:00
2022-07-25 23:47:17,508 DEBUG awx.main.dispatch task 5e088ad5-ca9e-4844-8a4e-8247bea8c15b is finished
2022-07-25 23:47:27,284 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(345066ae-7aed-4723-acae-ea77eb0bcc5f, queue=awx_private_queue)
[2022-07-25 23:47:27,284: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(345066ae-7aed-4723-acae-ea77eb0bcc5f, queue=awx_private_queue)
2022-07-25 23:47:27,302 DEBUG awx.main.dispatch delivered 345066ae-7aed-4723-acae-ea77eb0bcc5f to worker[180] qsize 0
2022-07-25 23:47:27,305 DEBUG awx.main.dispatch task 345066ae-7aed-4723-acae-ea77eb0bcc5f starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:47:27,306 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:47:27,425 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:47:27,574 DEBUG awx.main.dispatch task 345066ae-7aed-4723-acae-ea77eb0bcc5f is finished
2022-07-25 23:47:46,600 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(84382cce-a03e-4e97-bdea-281e50f28214, queue=onap-awx-0)
[2022-07-25 23:47:46,600: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(84382cce-a03e-4e97-bdea-281e50f28214, queue=onap-awx-0)
2022-07-25 23:47:46,706 DEBUG awx.main.dispatch delivered 84382cce-a03e-4e97-bdea-281e50f28214 to worker[181] qsize 0
2022-07-25 23:47:46,709 DEBUG awx.main.dispatch task 84382cce-a03e-4e97-bdea-281e50f28214 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:47:46,711 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:47:46,772 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(0053cd51-cdb3-404d-8a7b-03a3b8d3a8d0, queue=onap-awx-0)
[2022-07-25 23:47:46,772: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(0053cd51-cdb3-404d-8a7b-03a3b8d3a8d0, queue=onap-awx-0)
2022-07-25 23:47:46,789 DEBUG awx.main.dispatch delivered 0053cd51-cdb3-404d-8a7b-03a3b8d3a8d0 to worker[183] qsize 0
2022-07-25 23:47:46,791 DEBUG awx.main.dispatch task 0053cd51-cdb3-404d-8a7b-03a3b8d3a8d0 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:47:46,951 DEBUG awx.main.dispatch task 84382cce-a03e-4e97-bdea-281e50f28214 is finished
2022-07-25 23:47:46,986 DEBUG awx.main.dispatch task 0053cd51-cdb3-404d-8a7b-03a3b8d3a8d0 is finished
2022-07-25 23:47:47,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(028a44e2-c3e7-46eb-928e-e4e8916a50ab, queue=awx_private_queue)
[2022-07-25 23:47:47,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(028a44e2-c3e7-46eb-928e-e4e8916a50ab, queue=awx_private_queue)
2022-07-25 23:47:47,111 DEBUG awx.main.dispatch delivered 028a44e2-c3e7-46eb-928e-e4e8916a50ab to worker[181] qsize 0
2022-07-25 23:47:47,113 DEBUG awx.main.dispatch task 028a44e2-c3e7-46eb-928e-e4e8916a50ab starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:47:47,241 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:47:47,252 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:47:17.444706+00:00
2022-07-25 23:47:47,284 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a5915c76-2807-44d1-b36a-c1cbfede08c7, queue=awx_private_queue)
[2022-07-25 23:47:47,284: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a5915c76-2807-44d1-b36a-c1cbfede08c7, queue=awx_private_queue)
2022-07-25 23:47:47,294 DEBUG awx.main.dispatch task 028a44e2-c3e7-46eb-928e-e4e8916a50ab is finished
2022-07-25 23:47:47,305 DEBUG awx.main.dispatch delivered a5915c76-2807-44d1-b36a-c1cbfede08c7 to worker[180] qsize 0
2022-07-25 23:47:47,308 DEBUG awx.main.dispatch task a5915c76-2807-44d1-b36a-c1cbfede08c7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:47:47,310 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:47:47,483 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:47:47,837 DEBUG awx.main.dispatch task a5915c76-2807-44d1-b36a-c1cbfede08c7 is finished
RESULT 2
OKREADY
2022-07-25 23:48:07,284 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5becdda7-67ac-4904-a2db-37d6b744226a, queue=awx_private_queue)
[2022-07-25 23:48:07,284: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5becdda7-67ac-4904-a2db-37d6b744226a, queue=awx_private_queue)
2022-07-25 23:48:07,304 DEBUG awx.main.dispatch delivered 5becdda7-67ac-4904-a2db-37d6b744226a to worker[181] qsize 0
2022-07-25 23:48:07,307 DEBUG awx.main.dispatch task 5becdda7-67ac-4904-a2db-37d6b744226a starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:48:07,308 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:48:07,313 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:48:07,316 DEBUG awx.main.dispatch task 5becdda7-67ac-4904-a2db-37d6b744226a is finished
2022-07-25 23:48:17,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4e05d6d9-aa3c-487b-bcca-e6e5370493e8, queue=awx_private_queue)
[2022-07-25 23:48:17,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4e05d6d9-aa3c-487b-bcca-e6e5370493e8, queue=awx_private_queue)
2022-07-25 23:48:17,113 DEBUG awx.main.dispatch delivered 4e05d6d9-aa3c-487b-bcca-e6e5370493e8 to worker[183] qsize 0
2022-07-25 23:48:17,156 DEBUG awx.main.dispatch task 4e05d6d9-aa3c-487b-bcca-e6e5370493e8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:48:17,161 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:48:17,163 DEBUG awx.main.dispatch task 4e05d6d9-aa3c-487b-bcca-e6e5370493e8 is finished
2022-07-25 23:48:27,284 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2fc18e74-bddf-4a66-bf78-bd4bedb90a10, queue=awx_private_queue)
[2022-07-25 23:48:27,284: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2fc18e74-bddf-4a66-bf78-bd4bedb90a10, queue=awx_private_queue)
2022-07-25 23:48:27,301 DEBUG awx.main.dispatch delivered 2fc18e74-bddf-4a66-bf78-bd4bedb90a10 to worker[181] qsize 0
2022-07-25 23:48:27,546 DEBUG awx.main.dispatch task 2fc18e74-bddf-4a66-bf78-bd4bedb90a10 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:48:27,548 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:48:27,557 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:48:27,560 DEBUG awx.main.dispatch task 2fc18e74-bddf-4a66-bf78-bd4bedb90a10 is finished
2022-07-25 23:48:46,608 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(22536941-f4d8-41c2-995d-6deea12eb8da, queue=onap-awx-0)
[2022-07-25 23:48:46,608: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(22536941-f4d8-41c2-995d-6deea12eb8da, queue=onap-awx-0)
2022-07-25 23:48:46,657 DEBUG awx.main.dispatch delivered 22536941-f4d8-41c2-995d-6deea12eb8da to worker[182] qsize 0
2022-07-25 23:48:46,660 DEBUG awx.main.dispatch task 22536941-f4d8-41c2-995d-6deea12eb8da starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:48:46,662 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:48:46,668 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:48:46,670 DEBUG awx.main.dispatch task 22536941-f4d8-41c2-995d-6deea12eb8da is finished
2022-07-25 23:48:46,773 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(e0633c08-3709-45e5-bde0-19e65ce19ce2, queue=onap-awx-0)
[2022-07-25 23:48:46,773: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(e0633c08-3709-45e5-bde0-19e65ce19ce2, queue=onap-awx-0)
2022-07-25 23:48:46,790 DEBUG awx.main.dispatch delivered e0633c08-3709-45e5-bde0-19e65ce19ce2 to worker[183] qsize 0
2022-07-25 23:48:46,792 DEBUG awx.main.dispatch task e0633c08-3709-45e5-bde0-19e65ce19ce2 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:48:46,799 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:48:46,801 DEBUG awx.main.dispatch task e0633c08-3709-45e5-bde0-19e65ce19ce2 is finished
2022-07-25 23:48:47,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2ea07a52-cda8-4e83-a7db-ac4ec1f19825, queue=awx_private_queue)
[2022-07-25 23:48:47,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2ea07a52-cda8-4e83-a7db-ac4ec1f19825, queue=awx_private_queue)
2022-07-25 23:48:47,108 DEBUG awx.main.dispatch delivered 2ea07a52-cda8-4e83-a7db-ac4ec1f19825 to worker[183] qsize 0
2022-07-25 23:48:47,111 DEBUG awx.main.dispatch task 2ea07a52-cda8-4e83-a7db-ac4ec1f19825 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:48:47,115 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:48:47,118 DEBUG awx.main.dispatch task 2ea07a52-cda8-4e83-a7db-ac4ec1f19825 is finished
2022-07-25 23:48:47,284 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(543b12f4-0c50-4a79-9771-1af442747f87, queue=awx_private_queue)
[2022-07-25 23:48:47,284: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(543b12f4-0c50-4a79-9771-1af442747f87, queue=awx_private_queue)
2022-07-25 23:48:47,302 DEBUG awx.main.dispatch delivered 543b12f4-0c50-4a79-9771-1af442747f87 to worker[183] qsize 0
2022-07-25 23:48:47,305 DEBUG awx.main.dispatch task 543b12f4-0c50-4a79-9771-1af442747f87 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:48:47,307 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:48:47,310 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:48:47,312 DEBUG awx.main.dispatch task 543b12f4-0c50-4a79-9771-1af442747f87 is finished
RESULT 2
OKREADY
2022-07-25 23:49:07,292 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(58498bb2-f979-4510-905b-2c875f99b760, queue=awx_private_queue)
[2022-07-25 23:49:07,292: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(58498bb2-f979-4510-905b-2c875f99b760, queue=awx_private_queue)
2022-07-25 23:49:07,314 DEBUG awx.main.dispatch delivered 58498bb2-f979-4510-905b-2c875f99b760 to worker[181] qsize 0
2022-07-25 23:49:07,316 DEBUG awx.main.dispatch task 58498bb2-f979-4510-905b-2c875f99b760 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:49:07,319 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:49:07,323 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:49:07,325 DEBUG awx.main.dispatch task 58498bb2-f979-4510-905b-2c875f99b760 is finished
2022-07-25 23:49:17,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(86d7bf3c-a660-45c3-9188-2158d4ece530, queue=awx_private_queue)
[2022-07-25 23:49:17,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(86d7bf3c-a660-45c3-9188-2158d4ece530, queue=awx_private_queue)
2022-07-25 23:49:17,110 DEBUG awx.main.dispatch delivered 86d7bf3c-a660-45c3-9188-2158d4ece530 to worker[182] qsize 0
2022-07-25 23:49:17,112 DEBUG awx.main.dispatch task 86d7bf3c-a660-45c3-9188-2158d4ece530 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:49:17,172 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:49:17,183 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:47:47.242221+00:00
2022-07-25 23:49:17,288 DEBUG awx.main.dispatch task 86d7bf3c-a660-45c3-9188-2158d4ece530 is finished
2022-07-25 23:49:27,292 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2d22b01e-d01e-4497-bdcf-cf2ec30508b9, queue=awx_private_queue)
[2022-07-25 23:49:27,292: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2d22b01e-d01e-4497-bdcf-cf2ec30508b9, queue=awx_private_queue)
2022-07-25 23:49:27,995 DEBUG awx.main.dispatch delivered 2d22b01e-d01e-4497-bdcf-cf2ec30508b9 to worker[181] qsize 0
2022-07-25 23:49:28,439 DEBUG awx.main.dispatch task 2d22b01e-d01e-4497-bdcf-cf2ec30508b9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:49:28,442 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:49:29,663 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:49:30,203 DEBUG awx.main.dispatch task 2d22b01e-d01e-4497-bdcf-cf2ec30508b9 is finished
2022-07-25 23:49:46,617 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(37b46c29-6e4d-4b37-a144-2d7124cacfac, queue=onap-awx-0)
[2022-07-25 23:49:46,617: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(37b46c29-6e4d-4b37-a144-2d7124cacfac, queue=onap-awx-0)
2022-07-25 23:49:46,772 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(cc6ccbf8-1a04-4d3a-8a95-39275f4c7943, queue=onap-awx-0)
[2022-07-25 23:49:46,772: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(cc6ccbf8-1a04-4d3a-8a95-39275f4c7943, queue=onap-awx-0)
2022-07-25 23:49:46,813 DEBUG awx.main.dispatch delivered 37b46c29-6e4d-4b37-a144-2d7124cacfac to worker[181] qsize 0
2022-07-25 23:49:46,817 DEBUG awx.main.dispatch delivered cc6ccbf8-1a04-4d3a-8a95-39275f4c7943 to worker[180] qsize 0
2022-07-25 23:49:46,817 DEBUG awx.main.dispatch task 37b46c29-6e4d-4b37-a144-2d7124cacfac starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:49:46,820 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:49:46,819 DEBUG awx.main.dispatch task cc6ccbf8-1a04-4d3a-8a95-39275f4c7943 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:49:47,084 DEBUG awx.main.dispatch task cc6ccbf8-1a04-4d3a-8a95-39275f4c7943 is finished
2022-07-25 23:49:47,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7eac14b7-f6e2-4f50-b1e1-2f204a17acb6, queue=awx_private_queue)
[2022-07-25 23:49:47,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7eac14b7-f6e2-4f50-b1e1-2f204a17acb6, queue=awx_private_queue)
2022-07-25 23:49:47,113 DEBUG awx.main.dispatch delivered 7eac14b7-f6e2-4f50-b1e1-2f204a17acb6 to worker[180] qsize 0
2022-07-25 23:49:47,116 DEBUG awx.main.dispatch task 7eac14b7-f6e2-4f50-b1e1-2f204a17acb6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:49:47,292 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(30885b8d-0e0d-431f-a98e-50a389947364, queue=awx_private_queue)
[2022-07-25 23:49:47,292: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(30885b8d-0e0d-431f-a98e-50a389947364, queue=awx_private_queue)
2022-07-25 23:49:47,309 DEBUG awx.main.dispatch delivered 30885b8d-0e0d-431f-a98e-50a389947364 to worker[183] qsize 0
2022-07-25 23:49:47,312 DEBUG awx.main.dispatch task 30885b8d-0e0d-431f-a98e-50a389947364 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:49:47,315 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:49:47,364 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:49:47,473 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:49:17.172721+00:00
2022-07-25 23:49:47,653 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:49:47,862 DEBUG awx.main.dispatch task 7eac14b7-f6e2-4f50-b1e1-2f204a17acb6 is finished
2022-07-25 23:49:47,865 DEBUG awx.main.dispatch task 37b46c29-6e4d-4b37-a144-2d7124cacfac is finished
2022-07-25 23:49:47,962 DEBUG awx.main.dispatch task 30885b8d-0e0d-431f-a98e-50a389947364 is finished
RESULT 2
OKREADY
2022-07-25 23:50:07,302 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(dcb890b1-bae1-401c-9939-52f9c98d36f7, queue=awx_private_queue)
[2022-07-25 23:50:07,302: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(dcb890b1-bae1-401c-9939-52f9c98d36f7, queue=awx_private_queue)
2022-07-25 23:50:07,326 DEBUG awx.main.dispatch delivered dcb890b1-bae1-401c-9939-52f9c98d36f7 to worker[182] qsize 0
2022-07-25 23:50:07,330 DEBUG awx.main.dispatch task dcb890b1-bae1-401c-9939-52f9c98d36f7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:50:07,335 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:50:07,497 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:50:07,640 DEBUG awx.main.dispatch task dcb890b1-bae1-401c-9939-52f9c98d36f7 is finished
2022-07-25 23:50:17,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(30f4ca3c-b9d4-4297-8288-86d2b62e4626, queue=awx_private_queue)
[2022-07-25 23:50:17,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(30f4ca3c-b9d4-4297-8288-86d2b62e4626, queue=awx_private_queue)
2022-07-25 23:50:17,111 DEBUG awx.main.dispatch delivered 30f4ca3c-b9d4-4297-8288-86d2b62e4626 to worker[183] qsize 0
2022-07-25 23:50:17,114 DEBUG awx.main.dispatch task 30f4ca3c-b9d4-4297-8288-86d2b62e4626 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:50:17,438 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:50:17,467 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:49:47.365458+00:00
2022-07-25 23:50:17,671 DEBUG awx.main.dispatch task 30f4ca3c-b9d4-4297-8288-86d2b62e4626 is finished
2022-07-25 23:50:27,303 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e9776c99-5897-4b46-99f1-510fc2d29a62, queue=awx_private_queue)
[2022-07-25 23:50:27,303: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e9776c99-5897-4b46-99f1-510fc2d29a62, queue=awx_private_queue)
2022-07-25 23:50:27,322 DEBUG awx.main.dispatch delivered e9776c99-5897-4b46-99f1-510fc2d29a62 to worker[181] qsize 0
2022-07-25 23:50:27,324 DEBUG awx.main.dispatch task e9776c99-5897-4b46-99f1-510fc2d29a62 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:50:27,327 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:50:27,506 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:50:27,754 DEBUG awx.main.dispatch task e9776c99-5897-4b46-99f1-510fc2d29a62 is finished
2022-07-25 23:50:46,626 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c87519bb-65b6-4cc3-88e6-ec6657c39b84, queue=onap-awx-0)
[2022-07-25 23:50:46,626: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c87519bb-65b6-4cc3-88e6-ec6657c39b84, queue=onap-awx-0)
2022-07-25 23:50:46,772 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(fd3afd8a-aa56-45be-8c50-a77391bb3c26, queue=onap-awx-0)
[2022-07-25 23:50:46,772: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(fd3afd8a-aa56-45be-8c50-a77391bb3c26, queue=onap-awx-0)
2022-07-25 23:50:46,746 DEBUG awx.main.dispatch delivered c87519bb-65b6-4cc3-88e6-ec6657c39b84 to worker[180] qsize 0
2022-07-25 23:50:46,968 DEBUG awx.main.dispatch delivered fd3afd8a-aa56-45be-8c50-a77391bb3c26 to worker[182] qsize 0
2022-07-25 23:50:46,968 DEBUG awx.main.dispatch task c87519bb-65b6-4cc3-88e6-ec6657c39b84 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:50:46,971 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:50:46,970 DEBUG awx.main.dispatch task fd3afd8a-aa56-45be-8c50-a77391bb3c26 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:50:46,975 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:50:46,975 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:50:46,977 DEBUG awx.main.dispatch task fd3afd8a-aa56-45be-8c50-a77391bb3c26 is finished
2022-07-25 23:50:46,977 DEBUG awx.main.dispatch task c87519bb-65b6-4cc3-88e6-ec6657c39b84 is finished
2022-07-25 23:50:47,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(fba6c83b-d46b-4595-9e4c-b5f5c1d10946, queue=awx_private_queue)
[2022-07-25 23:50:47,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(fba6c83b-d46b-4595-9e4c-b5f5c1d10946, queue=awx_private_queue)
2022-07-25 23:50:47,109 DEBUG awx.main.dispatch delivered fba6c83b-d46b-4595-9e4c-b5f5c1d10946 to worker[181] qsize 0
2022-07-25 23:50:47,112 DEBUG awx.main.dispatch task fba6c83b-d46b-4595-9e4c-b5f5c1d10946 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:50:47,115 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:50:47,119 DEBUG awx.main.dispatch task fba6c83b-d46b-4595-9e4c-b5f5c1d10946 is finished
2022-07-25 23:50:47,303 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c8893dd6-e541-4390-9743-97038bc436f6, queue=awx_private_queue)
[2022-07-25 23:50:47,303: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c8893dd6-e541-4390-9743-97038bc436f6, queue=awx_private_queue)
2022-07-25 23:50:47,319 DEBUG awx.main.dispatch delivered c8893dd6-e541-4390-9743-97038bc436f6 to worker[183] qsize 0
2022-07-25 23:50:47,322 DEBUG awx.main.dispatch task c8893dd6-e541-4390-9743-97038bc436f6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:50:47,324 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:50:47,326 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:50:47,329 DEBUG awx.main.dispatch task c8893dd6-e541-4390-9743-97038bc436f6 is finished
RESULT 2
OKREADY
2022-07-25 23:51:07,307 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(65f68484-762c-4cc0-a5ff-f408dde877e8, queue=awx_private_queue)
[2022-07-25 23:51:07,307: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(65f68484-762c-4cc0-a5ff-f408dde877e8, queue=awx_private_queue)
2022-07-25 23:51:07,325 DEBUG awx.main.dispatch delivered 65f68484-762c-4cc0-a5ff-f408dde877e8 to worker[182] qsize 0
2022-07-25 23:51:07,327 DEBUG awx.main.dispatch task 65f68484-762c-4cc0-a5ff-f408dde877e8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:51:07,329 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:51:07,336 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:51:07,339 DEBUG awx.main.dispatch task 65f68484-762c-4cc0-a5ff-f408dde877e8 is finished
2022-07-25 23:51:17,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(afbb6525-9074-4c60-94c2-0a1be015c208, queue=awx_private_queue)
[2022-07-25 23:51:17,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(afbb6525-9074-4c60-94c2-0a1be015c208, queue=awx_private_queue)
2022-07-25 23:51:17,110 DEBUG awx.main.dispatch delivered afbb6525-9074-4c60-94c2-0a1be015c208 to worker[181] qsize 0
2022-07-25 23:51:17,112 DEBUG awx.main.dispatch task afbb6525-9074-4c60-94c2-0a1be015c208 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:51:17,118 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:51:17,121 DEBUG awx.main.dispatch task afbb6525-9074-4c60-94c2-0a1be015c208 is finished
2022-07-25 23:51:27,308 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f0baf3a5-bf19-4038-b158-eed31e405d05, queue=awx_private_queue)
[2022-07-25 23:51:27,308: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f0baf3a5-bf19-4038-b158-eed31e405d05, queue=awx_private_queue)
2022-07-25 23:51:27,326 DEBUG awx.main.dispatch delivered f0baf3a5-bf19-4038-b158-eed31e405d05 to worker[183] qsize 0
2022-07-25 23:51:27,328 DEBUG awx.main.dispatch task f0baf3a5-bf19-4038-b158-eed31e405d05 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:51:27,330 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:51:27,334 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:51:27,336 DEBUG awx.main.dispatch task f0baf3a5-bf19-4038-b158-eed31e405d05 is finished
2022-07-25 23:51:46,628 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(01b8ea04-cf78-4c0b-a026-96086689146a, queue=onap-awx-0)
[2022-07-25 23:51:46,628: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(01b8ea04-cf78-4c0b-a026-96086689146a, queue=onap-awx-0)
2022-07-25 23:51:46,742 DEBUG awx.main.dispatch delivered 01b8ea04-cf78-4c0b-a026-96086689146a to worker[181] qsize 0
2022-07-25 23:51:46,773 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(001c2397-e531-4493-8840-c7fc7eba492c, queue=onap-awx-0)
[2022-07-25 23:51:46,773: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(001c2397-e531-4493-8840-c7fc7eba492c, queue=onap-awx-0)
2022-07-25 23:51:46,774 DEBUG awx.main.dispatch task 01b8ea04-cf78-4c0b-a026-96086689146a starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:51:46,784 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:51:46,787 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:51:46,790 DEBUG awx.main.dispatch task 01b8ea04-cf78-4c0b-a026-96086689146a is finished
2022-07-25 23:51:46,790 DEBUG awx.main.dispatch delivered 001c2397-e531-4493-8840-c7fc7eba492c to worker[180] qsize 0
2022-07-25 23:51:47,004 DEBUG awx.main.dispatch task 001c2397-e531-4493-8840-c7fc7eba492c starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:51:47,011 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:51:47,014 DEBUG awx.main.dispatch task 001c2397-e531-4493-8840-c7fc7eba492c is finished
2022-07-25 23:51:47,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3ef5ef70-e4a3-40f7-9d7a-a6fee240b894, queue=awx_private_queue)
[2022-07-25 23:51:47,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3ef5ef70-e4a3-40f7-9d7a-a6fee240b894, queue=awx_private_queue)
2022-07-25 23:51:47,111 DEBUG awx.main.dispatch delivered 3ef5ef70-e4a3-40f7-9d7a-a6fee240b894 to worker[182] qsize 0
2022-07-25 23:51:47,114 DEBUG awx.main.dispatch task 3ef5ef70-e4a3-40f7-9d7a-a6fee240b894 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:51:47,118 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:51:47,120 DEBUG awx.main.dispatch task 3ef5ef70-e4a3-40f7-9d7a-a6fee240b894 is finished
2022-07-25 23:51:47,307 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(72cf0aac-d88f-4cd1-95ea-80408f0193f7, queue=awx_private_queue)
[2022-07-25 23:51:47,307: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(72cf0aac-d88f-4cd1-95ea-80408f0193f7, queue=awx_private_queue)
2022-07-25 23:51:47,325 DEBUG awx.main.dispatch delivered 72cf0aac-d88f-4cd1-95ea-80408f0193f7 to worker[181] qsize 0
2022-07-25 23:51:47,328 DEBUG awx.main.dispatch task 72cf0aac-d88f-4cd1-95ea-80408f0193f7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:51:47,330 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:51:47,332 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:51:47,335 DEBUG awx.main.dispatch task 72cf0aac-d88f-4cd1-95ea-80408f0193f7 is finished
RESULT 2
OKREADY
2022-07-25 23:52:07,309 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(aea23d78-a1c3-4d54-8d61-fc7c8964f813, queue=awx_private_queue)
[2022-07-25 23:52:07,309: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(aea23d78-a1c3-4d54-8d61-fc7c8964f813, queue=awx_private_queue)
2022-07-25 23:52:07,333 DEBUG awx.main.dispatch delivered aea23d78-a1c3-4d54-8d61-fc7c8964f813 to worker[182] qsize 0
2022-07-25 23:52:07,336 DEBUG awx.main.dispatch task aea23d78-a1c3-4d54-8d61-fc7c8964f813 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:52:07,338 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:52:07,343 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:52:07,348 DEBUG awx.main.dispatch task aea23d78-a1c3-4d54-8d61-fc7c8964f813 is finished
2022-07-25 23:52:17,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(239fad0e-e657-4ff4-98c1-71ddbf2dea74, queue=awx_private_queue)
[2022-07-25 23:52:17,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(239fad0e-e657-4ff4-98c1-71ddbf2dea74, queue=awx_private_queue)
2022-07-25 23:52:17,116 DEBUG awx.main.dispatch delivered 239fad0e-e657-4ff4-98c1-71ddbf2dea74 to worker[180] qsize 0
2022-07-25 23:52:17,119 DEBUG awx.main.dispatch task 239fad0e-e657-4ff4-98c1-71ddbf2dea74 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:52:17,124 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:52:17,128 DEBUG awx.main.dispatch task 239fad0e-e657-4ff4-98c1-71ddbf2dea74 is finished
2022-07-25 23:52:27,310 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ad8d3a6e-7133-48ae-8e8c-592449e4ce02, queue=awx_private_queue)
[2022-07-25 23:52:27,310: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ad8d3a6e-7133-48ae-8e8c-592449e4ce02, queue=awx_private_queue)
2022-07-25 23:52:27,335 DEBUG awx.main.dispatch delivered ad8d3a6e-7133-48ae-8e8c-592449e4ce02 to worker[180] qsize 0
2022-07-25 23:52:27,338 DEBUG awx.main.dispatch task ad8d3a6e-7133-48ae-8e8c-592449e4ce02 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:52:27,339 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:52:27,344 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:52:27,346 DEBUG awx.main.dispatch task ad8d3a6e-7133-48ae-8e8c-592449e4ce02 is finished
2022-07-25 23:52:46,638 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e37693df-93e8-4fa5-96af-20de6ff593c9, queue=onap-awx-0)
[2022-07-25 23:52:46,638: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e37693df-93e8-4fa5-96af-20de6ff593c9, queue=onap-awx-0)
2022-07-25 23:52:46,741 DEBUG awx.main.dispatch delivered e37693df-93e8-4fa5-96af-20de6ff593c9 to worker[180] qsize 0
2022-07-25 23:52:46,762 DEBUG awx.main.dispatch task e37693df-93e8-4fa5-96af-20de6ff593c9 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:52:46,763 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:52:46,768 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:52:46,771 DEBUG awx.main.dispatch task e37693df-93e8-4fa5-96af-20de6ff593c9 is finished
2022-07-25 23:52:46,773 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3aac766e-a08c-4316-bb1f-8f550159d6f2, queue=onap-awx-0)
[2022-07-25 23:52:46,773: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3aac766e-a08c-4316-bb1f-8f550159d6f2, queue=onap-awx-0)
2022-07-25 23:52:46,791 DEBUG awx.main.dispatch delivered 3aac766e-a08c-4316-bb1f-8f550159d6f2 to worker[182] qsize 0
2022-07-25 23:52:47,020 DEBUG awx.main.dispatch task 3aac766e-a08c-4316-bb1f-8f550159d6f2 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:52:47,025 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:52:47,027 DEBUG awx.main.dispatch task 3aac766e-a08c-4316-bb1f-8f550159d6f2 is finished
2022-07-25 23:52:47,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4e31ebc2-1f3f-4c43-907b-eb6423378a59, queue=awx_private_queue)
[2022-07-25 23:52:47,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4e31ebc2-1f3f-4c43-907b-eb6423378a59, queue=awx_private_queue)
2022-07-25 23:52:47,111 DEBUG awx.main.dispatch delivered 4e31ebc2-1f3f-4c43-907b-eb6423378a59 to worker[180] qsize 0
2022-07-25 23:52:47,115 DEBUG awx.main.dispatch task 4e31ebc2-1f3f-4c43-907b-eb6423378a59 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:52:47,120 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:52:47,124 DEBUG awx.main.dispatch task 4e31ebc2-1f3f-4c43-907b-eb6423378a59 is finished
2022-07-25 23:52:47,309 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0faa6211-1f53-4646-b10b-62af9ab98893, queue=awx_private_queue)
[2022-07-25 23:52:47,309: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0faa6211-1f53-4646-b10b-62af9ab98893, queue=awx_private_queue)
2022-07-25 23:52:47,327 DEBUG awx.main.dispatch delivered 0faa6211-1f53-4646-b10b-62af9ab98893 to worker[183] qsize 0
2022-07-25 23:52:47,330 DEBUG awx.main.dispatch task 0faa6211-1f53-4646-b10b-62af9ab98893 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:52:47,332 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:52:47,335 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:52:47,336 DEBUG awx.main.dispatch task 0faa6211-1f53-4646-b10b-62af9ab98893 is finished
RESULT 2
OKREADY
2022-07-25 23:53:07,324 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b8deddbc-2b7c-4071-820b-80551aa204a6, queue=awx_private_queue)
[2022-07-25 23:53:07,324: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b8deddbc-2b7c-4071-820b-80551aa204a6, queue=awx_private_queue)
2022-07-25 23:53:07,349 DEBUG awx.main.dispatch delivered b8deddbc-2b7c-4071-820b-80551aa204a6 to worker[180] qsize 0
2022-07-25 23:53:07,352 DEBUG awx.main.dispatch task b8deddbc-2b7c-4071-820b-80551aa204a6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:53:07,355 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:53:07,360 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:53:07,363 DEBUG awx.main.dispatch task b8deddbc-2b7c-4071-820b-80551aa204a6 is finished
2022-07-25 23:53:17,093 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3782e7a2-911e-4024-8d3d-b17c08422c03, queue=awx_private_queue)
[2022-07-25 23:53:17,093: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3782e7a2-911e-4024-8d3d-b17c08422c03, queue=awx_private_queue)
2022-07-25 23:53:17,110 DEBUG awx.main.dispatch delivered 3782e7a2-911e-4024-8d3d-b17c08422c03 to worker[182] qsize 0
2022-07-25 23:53:17,113 DEBUG awx.main.dispatch task 3782e7a2-911e-4024-8d3d-b17c08422c03 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:53:17,176 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:53:17,187 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:50:17.438776+00:00
2022-07-25 23:53:17,314 DEBUG awx.main.dispatch task 3782e7a2-911e-4024-8d3d-b17c08422c03 is finished
2022-07-25 23:53:27,324 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(638801d4-6bfe-43f9-9dbe-896b5ce8496a, queue=awx_private_queue)
[2022-07-25 23:53:27,324: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(638801d4-6bfe-43f9-9dbe-896b5ce8496a, queue=awx_private_queue)
2022-07-25 23:53:27,345 DEBUG awx.main.dispatch delivered 638801d4-6bfe-43f9-9dbe-896b5ce8496a to worker[181] qsize 0
2022-07-25 23:53:27,348 DEBUG awx.main.dispatch task 638801d4-6bfe-43f9-9dbe-896b5ce8496a starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:53:27,352 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:53:27,820 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:53:28,227 DEBUG awx.main.dispatch task 638801d4-6bfe-43f9-9dbe-896b5ce8496a is finished
2022-07-25 23:53:46,647 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(15408349-3d99-4b01-be80-0f331ea6d49a, queue=onap-awx-0)
[2022-07-25 23:53:46,647: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(15408349-3d99-4b01-be80-0f331ea6d49a, queue=onap-awx-0)
2022-07-25 23:53:46,791 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3a1954ba-85fc-4c34-9f5b-3c221c94ac4f, queue=onap-awx-0)
[2022-07-25 23:53:46,791: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3a1954ba-85fc-4c34-9f5b-3c221c94ac4f, queue=onap-awx-0)
2022-07-25 23:53:47,093 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(01ea6dd1-b65f-43d5-9fcb-fff257780589, queue=awx_private_queue)
[2022-07-25 23:53:47,093: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(01ea6dd1-b65f-43d5-9fcb-fff257780589, queue=awx_private_queue)
2022-07-25 23:53:46,887 DEBUG awx.main.dispatch delivered 15408349-3d99-4b01-be80-0f331ea6d49a to worker[183] qsize 0
2022-07-25 23:53:47,139 DEBUG awx.main.dispatch delivered 3a1954ba-85fc-4c34-9f5b-3c221c94ac4f to worker[182] qsize 0
2022-07-25 23:53:47,140 DEBUG awx.main.dispatch delivered 01ea6dd1-b65f-43d5-9fcb-fff257780589 to worker[180] qsize 0
2022-07-25 23:53:47,140 DEBUG awx.main.dispatch task 3a1954ba-85fc-4c34-9f5b-3c221c94ac4f starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:53:47,139 DEBUG awx.main.dispatch task 15408349-3d99-4b01-be80-0f331ea6d49a starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:53:47,143 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:53:47,142 DEBUG awx.main.dispatch task 01ea6dd1-b65f-43d5-9fcb-fff257780589 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:53:47,215 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:53:47,254 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:53:17.176639+00:00
2022-07-25 23:53:47,323 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(201d0a77-c6bf-4561-98b2-77bf668c382f, queue=awx_private_queue)
[2022-07-25 23:53:47,323: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(201d0a77-c6bf-4561-98b2-77bf668c382f, queue=awx_private_queue)
2022-07-25 23:53:47,334 DEBUG awx.main.dispatch task 3a1954ba-85fc-4c34-9f5b-3c221c94ac4f is finished
2022-07-25 23:53:47,343 DEBUG awx.main.dispatch delivered 201d0a77-c6bf-4561-98b2-77bf668c382f to worker[182] qsize 0
2022-07-25 23:53:47,345 DEBUG awx.main.dispatch task 201d0a77-c6bf-4561-98b2-77bf668c382f starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:53:47,347 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:53:47,407 WARNING awx.main.tasks Rejoining the cluster as instance onap-awx-0.
2022-07-25 23:53:47,407 DEBUG awx.main.dispatch task 15408349-3d99-4b01-be80-0f331ea6d49a is finished
2022-07-25 23:53:47,540 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:53:47,540 DEBUG awx.main.dispatch task 01ea6dd1-b65f-43d5-9fcb-fff257780589 is finished
2022-07-25 23:53:47,818 DEBUG awx.main.dispatch task 201d0a77-c6bf-4561-98b2-77bf668c382f is finished
RESULT 2
OKREADY
2022-07-25 23:54:07,327 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ffa3619f-317b-4540-9fce-3972c0382170, queue=awx_private_queue)
[2022-07-25 23:54:07,327: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ffa3619f-317b-4540-9fce-3972c0382170, queue=awx_private_queue)
2022-07-25 23:54:07,345 DEBUG awx.main.dispatch delivered ffa3619f-317b-4540-9fce-3972c0382170 to worker[182] qsize 0
2022-07-25 23:54:07,347 DEBUG awx.main.dispatch task ffa3619f-317b-4540-9fce-3972c0382170 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:54:07,348 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:54:07,440 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:54:07,639 DEBUG awx.main.dispatch task ffa3619f-317b-4540-9fce-3972c0382170 is finished
2022-07-25 23:54:17,093 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(026f18e2-14d6-447b-98f7-85db1be27a87, queue=awx_private_queue)
[2022-07-25 23:54:17,093: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(026f18e2-14d6-447b-98f7-85db1be27a87, queue=awx_private_queue)
2022-07-25 23:54:17,110 DEBUG awx.main.dispatch delivered 026f18e2-14d6-447b-98f7-85db1be27a87 to worker[181] qsize 0
2022-07-25 23:54:17,114 DEBUG awx.main.dispatch task 026f18e2-14d6-447b-98f7-85db1be27a87 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:54:17,216 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:54:17,236 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:53:47.215498+00:00
2022-07-25 23:54:17,378 DEBUG awx.main.dispatch task 026f18e2-14d6-447b-98f7-85db1be27a87 is finished
2022-07-25 23:54:27,327 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(97328ac9-e95f-4133-970a-391f59d8ceaf, queue=awx_private_queue)
[2022-07-25 23:54:27,327: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(97328ac9-e95f-4133-970a-391f59d8ceaf, queue=awx_private_queue)
2022-07-25 23:54:27,346 DEBUG awx.main.dispatch delivered 97328ac9-e95f-4133-970a-391f59d8ceaf to worker[182] qsize 0
2022-07-25 23:54:27,348 DEBUG awx.main.dispatch task 97328ac9-e95f-4133-970a-391f59d8ceaf starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:54:27,351 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:54:28,331 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:54:29,243 DEBUG awx.main.dispatch task 97328ac9-e95f-4133-970a-391f59d8ceaf is finished
2022-07-25 23:54:46,657 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(97709d55-81d1-45a5-8b12-711f21d9f454, queue=onap-awx-0)
[2022-07-25 23:54:46,657: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(97709d55-81d1-45a5-8b12-711f21d9f454, queue=onap-awx-0)
2022-07-25 23:54:46,808 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(dd7cd13d-3da9-45fa-9035-b9edb7ffa09e, queue=onap-awx-0)
[2022-07-25 23:54:46,808: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(dd7cd13d-3da9-45fa-9035-b9edb7ffa09e, queue=onap-awx-0)
2022-07-25 23:54:47,093 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bc74fe97-f7c9-43f4-80f2-d9fabadc112f, queue=awx_private_queue)
[2022-07-25 23:54:47,093: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bc74fe97-f7c9-43f4-80f2-d9fabadc112f, queue=awx_private_queue)
2022-07-25 23:54:46,931 DEBUG awx.main.dispatch delivered 97709d55-81d1-45a5-8b12-711f21d9f454 to worker[180] qsize 0
2022-07-25 23:54:47,186 DEBUG awx.main.dispatch delivered dd7cd13d-3da9-45fa-9035-b9edb7ffa09e to worker[181] qsize 0
2022-07-25 23:54:47,186 DEBUG awx.main.dispatch task 97709d55-81d1-45a5-8b12-711f21d9f454 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:54:47,188 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:54:47,188 DEBUG awx.main.dispatch delivered bc74fe97-f7c9-43f4-80f2-d9fabadc112f to worker[183] qsize 0
2022-07-25 23:54:47,188 DEBUG awx.main.dispatch task dd7cd13d-3da9-45fa-9035-b9edb7ffa09e starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:54:47,190 DEBUG awx.main.dispatch task bc74fe97-f7c9-43f4-80f2-d9fabadc112f starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:54:47,242 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:54:47,253 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:54:17.217180+00:00
2022-07-25 23:54:47,273 DEBUG awx.main.dispatch task dd7cd13d-3da9-45fa-9035-b9edb7ffa09e is finished
2022-07-25 23:54:47,327 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b031ecdb-6076-4e58-84ee-6e676b3b6a5f, queue=awx_private_queue)
[2022-07-25 23:54:47,327: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b031ecdb-6076-4e58-84ee-6e676b3b6a5f, queue=awx_private_queue)
2022-07-25 23:54:47,346 DEBUG awx.main.dispatch delivered b031ecdb-6076-4e58-84ee-6e676b3b6a5f to worker[181] qsize 0
2022-07-25 23:54:47,350 DEBUG awx.main.dispatch task b031ecdb-6076-4e58-84ee-6e676b3b6a5f starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:54:47,352 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:54:47,354 DEBUG awx.main.dispatch task 97709d55-81d1-45a5-8b12-711f21d9f454 is finished
2022-07-25 23:54:47,361 DEBUG awx.main.dispatch task bc74fe97-f7c9-43f4-80f2-d9fabadc112f is finished
2022-07-25 23:54:47,484 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:54:47,667 DEBUG awx.main.dispatch task b031ecdb-6076-4e58-84ee-6e676b3b6a5f is finished
RESULT 2
OKREADY
2022-07-25 23:55:07,335 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d7c6fbd3-fbd6-4b1b-aa4a-367494cc2974, queue=awx_private_queue)
[2022-07-25 23:55:07,335: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d7c6fbd3-fbd6-4b1b-aa4a-367494cc2974, queue=awx_private_queue)
2022-07-25 23:55:07,352 DEBUG awx.main.dispatch delivered d7c6fbd3-fbd6-4b1b-aa4a-367494cc2974 to worker[180] qsize 0
2022-07-25 23:55:07,355 DEBUG awx.main.dispatch task d7c6fbd3-fbd6-4b1b-aa4a-367494cc2974 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:55:07,356 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:55:07,438 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:55:07,604 DEBUG awx.main.dispatch task d7c6fbd3-fbd6-4b1b-aa4a-367494cc2974 is finished
2022-07-25 23:55:17,093 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(875fbcee-f241-4c94-8f1b-11dabad5270f, queue=awx_private_queue)
[2022-07-25 23:55:17,093: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(875fbcee-f241-4c94-8f1b-11dabad5270f, queue=awx_private_queue)
2022-07-25 23:55:17,111 DEBUG awx.main.dispatch delivered 875fbcee-f241-4c94-8f1b-11dabad5270f to worker[181] qsize 0
2022-07-25 23:55:17,113 DEBUG awx.main.dispatch task 875fbcee-f241-4c94-8f1b-11dabad5270f starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:55:17,135 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:55:17,140 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:54:47.242377+00:00
2022-07-25 23:55:17,166 DEBUG awx.main.dispatch task 875fbcee-f241-4c94-8f1b-11dabad5270f is finished
2022-07-25 23:55:27,336 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(257147a3-1e45-44ba-b276-13cd2cf6fe61, queue=awx_private_queue)
[2022-07-25 23:55:27,336: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(257147a3-1e45-44ba-b276-13cd2cf6fe61, queue=awx_private_queue)
2022-07-25 23:55:27,356 DEBUG awx.main.dispatch delivered 257147a3-1e45-44ba-b276-13cd2cf6fe61 to worker[180] qsize 0
2022-07-25 23:55:27,358 DEBUG awx.main.dispatch task 257147a3-1e45-44ba-b276-13cd2cf6fe61 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:55:27,360 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:55:27,518 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:55:27,709 DEBUG awx.main.dispatch task 257147a3-1e45-44ba-b276-13cd2cf6fe61 is finished
2022-07-25 23:55:46,667 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(2f8f759e-9e71-43ba-aff9-97d5074da002, queue=onap-awx-0)
[2022-07-25 23:55:46,667: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(2f8f759e-9e71-43ba-aff9-97d5074da002, queue=onap-awx-0)
2022-07-25 23:55:46,808 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5b2a4f6f-d07c-4f04-9301-9ad926fd6624, queue=onap-awx-0)
[2022-07-25 23:55:46,808: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5b2a4f6f-d07c-4f04-9301-9ad926fd6624, queue=onap-awx-0)
2022-07-25 23:55:46,837 DEBUG awx.main.dispatch delivered 2f8f759e-9e71-43ba-aff9-97d5074da002 to worker[182] qsize 0
2022-07-25 23:55:46,840 DEBUG awx.main.dispatch delivered 5b2a4f6f-d07c-4f04-9301-9ad926fd6624 to worker[180] qsize 0
2022-07-25 23:55:46,843 DEBUG awx.main.dispatch task 5b2a4f6f-d07c-4f04-9301-9ad926fd6624 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:55:46,928 DEBUG awx.main.dispatch task 5b2a4f6f-d07c-4f04-9301-9ad926fd6624 is finished
2022-07-25 23:55:47,093 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0180b5c1-2ff6-4a7e-9c5c-a0c33bb74f1e, queue=awx_private_queue)
[2022-07-25 23:55:47,093: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0180b5c1-2ff6-4a7e-9c5c-a0c33bb74f1e, queue=awx_private_queue)
2022-07-25 23:55:46,840 DEBUG awx.main.dispatch task 2f8f759e-9e71-43ba-aff9-97d5074da002 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:55:47,142 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:55:47,294 DEBUG awx.main.dispatch task 2f8f759e-9e71-43ba-aff9-97d5074da002 is finished
2022-07-25 23:55:47,335 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(37f456c7-a676-4f87-82fb-7fb8db62c977, queue=awx_private_queue)
[2022-07-25 23:55:47,335: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(37f456c7-a676-4f87-82fb-7fb8db62c977, queue=awx_private_queue)
2022-07-25 23:55:47,110 DEBUG awx.main.dispatch delivered 0180b5c1-2ff6-4a7e-9c5c-a0c33bb74f1e to worker[183] qsize 0
2022-07-25 23:55:47,350 DEBUG awx.main.dispatch task 0180b5c1-2ff6-4a7e-9c5c-a0c33bb74f1e starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:55:47,355 DEBUG awx.main.dispatch delivered 37f456c7-a676-4f87-82fb-7fb8db62c977 to worker[182] qsize 0
2022-07-25 23:55:47,358 DEBUG awx.main.dispatch task 37f456c7-a676-4f87-82fb-7fb8db62c977 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:55:47,360 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:55:47,520 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:55:47,545 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:55:17.136064+00:00
2022-07-25 23:55:47,560 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:55:47,694 DEBUG awx.main.dispatch task 0180b5c1-2ff6-4a7e-9c5c-a0c33bb74f1e is finished
2022-07-25 23:55:47,709 DEBUG awx.main.dispatch task 37f456c7-a676-4f87-82fb-7fb8db62c977 is finished
RESULT 2
OKREADY
2022-07-25 23:56:07,347 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d231f5f0-e4ee-4ecc-9bd1-513b7d50a762, queue=awx_private_queue)
[2022-07-25 23:56:07,347: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d231f5f0-e4ee-4ecc-9bd1-513b7d50a762, queue=awx_private_queue)
2022-07-25 23:56:07,377 DEBUG awx.main.dispatch delivered d231f5f0-e4ee-4ecc-9bd1-513b7d50a762 to worker[180] qsize 0
2022-07-25 23:56:07,382 DEBUG awx.main.dispatch task d231f5f0-e4ee-4ecc-9bd1-513b7d50a762 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:56:07,386 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:56:07,544 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:56:07,773 DEBUG awx.main.dispatch task d231f5f0-e4ee-4ecc-9bd1-513b7d50a762 is finished
2022-07-25 23:56:17,095 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(70787a29-76e1-4259-97f1-d9114d466102, queue=awx_private_queue)
[2022-07-25 23:56:17,095: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(70787a29-76e1-4259-97f1-d9114d466102, queue=awx_private_queue)
2022-07-25 23:56:17,115 DEBUG awx.main.dispatch delivered 70787a29-76e1-4259-97f1-d9114d466102 to worker[180] qsize 0
2022-07-25 23:56:17,118 DEBUG awx.main.dispatch task 70787a29-76e1-4259-97f1-d9114d466102 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:56:17,123 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:56:17,126 DEBUG awx.main.dispatch task 70787a29-76e1-4259-97f1-d9114d466102 is finished
2022-07-25 23:56:27,346 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c6d065d4-b24e-4a49-972e-8005a742bf74, queue=awx_private_queue)
[2022-07-25 23:56:27,346: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c6d065d4-b24e-4a49-972e-8005a742bf74, queue=awx_private_queue)
2022-07-25 23:56:27,380 DEBUG awx.main.dispatch delivered c6d065d4-b24e-4a49-972e-8005a742bf74 to worker[180] qsize 0
2022-07-25 23:56:27,382 DEBUG awx.main.dispatch task c6d065d4-b24e-4a49-972e-8005a742bf74 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:56:27,383 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:56:27,387 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-25 23:56:27,390 DEBUG awx.main.dispatch task c6d065d4-b24e-4a49-972e-8005a742bf74 is finished
2022-07-25 23:56:46,415 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(f0cebed8-daed-4581-92e0-4ff29a4f03c2, queue=onap-awx-0)
[2022-07-25 23:56:46,415: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(f0cebed8-daed-4581-92e0-4ff29a4f03c2, queue=onap-awx-0)
2022-07-25 23:56:46,434 DEBUG awx.main.dispatch delivered f0cebed8-daed-4581-92e0-4ff29a4f03c2 to worker[182] qsize 0
2022-07-25 23:56:46,437 DEBUG awx.main.dispatch task f0cebed8-daed-4581-92e0-4ff29a4f03c2 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2022-07-25 23:56:46,439 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2022-07-25 23:56:46,530 DEBUG awx.main.dispatch task f0cebed8-daed-4581-92e0-4ff29a4f03c2 is finished
2022-07-25 23:56:46,665 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d8703262-35ac-4761-8518-05dd03f83eef, queue=onap-awx-0)
[2022-07-25 23:56:46,665: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d8703262-35ac-4761-8518-05dd03f83eef, queue=onap-awx-0)
2022-07-25 23:56:46,805 DEBUG awx.main.dispatch delivered d8703262-35ac-4761-8518-05dd03f83eef to worker[182] qsize 0
2022-07-25 23:56:46,808 DEBUG awx.main.dispatch task d8703262-35ac-4761-8518-05dd03f83eef starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:56:46,810 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:56:46,809 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(fc8b96b5-1187-4a33-a085-52f3bc8661f2, queue=onap-awx-0)
[2022-07-25 23:56:46,809: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(fc8b96b5-1187-4a33-a085-52f3bc8661f2, queue=onap-awx-0)
2022-07-25 23:56:46,824 DEBUG awx.main.dispatch delivered fc8b96b5-1187-4a33-a085-52f3bc8661f2 to worker[183] qsize 0
2022-07-25 23:56:46,827 DEBUG awx.main.dispatch task fc8b96b5-1187-4a33-a085-52f3bc8661f2 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:56:46,971 DEBUG awx.main.dispatch task fc8b96b5-1187-4a33-a085-52f3bc8661f2 is finished
2022-07-25 23:56:47,095 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(be2f3358-47b3-4aaa-a33e-802c60d932cf, queue=awx_private_queue)
[2022-07-25 23:56:47,095: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(be2f3358-47b3-4aaa-a33e-802c60d932cf, queue=awx_private_queue)
2022-07-25 23:56:46,999 DEBUG awx.main.dispatch task d8703262-35ac-4761-8518-05dd03f83eef is finished
2022-07-25 23:56:47,111 DEBUG awx.main.dispatch delivered be2f3358-47b3-4aaa-a33e-802c60d932cf to worker[181] qsize 0
2022-07-25 23:56:47,329 DEBUG awx.main.dispatch task be2f3358-47b3-4aaa-a33e-802c60d932cf starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:56:47,346 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3fc52d73-6c42-41c2-8d7d-fcc0054d73f3, queue=awx_private_queue)
[2022-07-25 23:56:47,346: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3fc52d73-6c42-41c2-8d7d-fcc0054d73f3, queue=awx_private_queue)
2022-07-25 23:56:47,367 DEBUG awx.main.dispatch delivered 3fc52d73-6c42-41c2-8d7d-fcc0054d73f3 to worker[182] qsize 0
2022-07-25 23:56:47,370 DEBUG awx.main.dispatch task 3fc52d73-6c42-41c2-8d7d-fcc0054d73f3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:56:47,372 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:56:47,372 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:56:47,389 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:55:47.520956+00:00
2022-07-25 23:56:47,504 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:56:47,503 DEBUG awx.main.dispatch task be2f3358-47b3-4aaa-a33e-802c60d932cf is finished
2022-07-25 23:56:47,690 DEBUG awx.main.dispatch task 3fc52d73-6c42-41c2-8d7d-fcc0054d73f3 is finished
RESULT 2
OKREADY
2022-07-25 23:57:07,355 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(17d79521-e114-4246-b51d-c75939254768, queue=awx_private_queue)
[2022-07-25 23:57:07,355: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(17d79521-e114-4246-b51d-c75939254768, queue=awx_private_queue)
2022-07-25 23:57:07,378 DEBUG awx.main.dispatch delivered 17d79521-e114-4246-b51d-c75939254768 to worker[182] qsize 0
2022-07-25 23:57:07,380 DEBUG awx.main.dispatch task 17d79521-e114-4246-b51d-c75939254768 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:57:07,382 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:57:07,503 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:57:07,850 DEBUG awx.main.dispatch task 17d79521-e114-4246-b51d-c75939254768 is finished
2022-07-25 23:57:17,094 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7ddeaa0d-4460-48e3-b27c-54d24c2e0800, queue=awx_private_queue)
[2022-07-25 23:57:17,094: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7ddeaa0d-4460-48e3-b27c-54d24c2e0800, queue=awx_private_queue)
2022-07-25 23:57:17,134 DEBUG awx.main.dispatch delivered 7ddeaa0d-4460-48e3-b27c-54d24c2e0800 to worker[183] qsize 0
2022-07-25 23:57:17,137 DEBUG awx.main.dispatch task 7ddeaa0d-4460-48e3-b27c-54d24c2e0800 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:57:17,166 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:57:17,171 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:56:47.373536+00:00
2022-07-25 23:57:17,231 DEBUG awx.main.dispatch task 7ddeaa0d-4460-48e3-b27c-54d24c2e0800 is finished
2022-07-25 23:57:27,356 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ca89fedc-1e68-450e-933d-72240f67c7fb, queue=awx_private_queue)
[2022-07-25 23:57:27,356: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ca89fedc-1e68-450e-933d-72240f67c7fb, queue=awx_private_queue)
2022-07-25 23:57:27,376 DEBUG awx.main.dispatch delivered ca89fedc-1e68-450e-933d-72240f67c7fb to worker[182] qsize 0
2022-07-25 23:57:27,379 DEBUG awx.main.dispatch task ca89fedc-1e68-450e-933d-72240f67c7fb starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:57:27,380 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:57:27,852 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:57:28,179 DEBUG awx.main.dispatch task ca89fedc-1e68-450e-933d-72240f67c7fb is finished
2022-07-25 23:57:46,674 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(686ef635-5dd6-4476-8cd5-f981924ae9fa, queue=onap-awx-0)
[2022-07-25 23:57:46,674: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(686ef635-5dd6-4476-8cd5-f981924ae9fa, queue=onap-awx-0)
2022-07-25 23:57:46,902 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(215ab6a4-4008-45d1-8290-875d79f37c01, queue=onap-awx-0)
[2022-07-25 23:57:46,902: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(215ab6a4-4008-45d1-8290-875d79f37c01, queue=onap-awx-0)
2022-07-25 23:57:46,989 DEBUG awx.main.dispatch delivered 686ef635-5dd6-4476-8cd5-f981924ae9fa to worker[183] qsize 0
2022-07-25 23:57:47,000 DEBUG awx.main.dispatch task 686ef635-5dd6-4476-8cd5-f981924ae9fa starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:57:47,001 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:57:47,001 DEBUG awx.main.dispatch delivered 215ab6a4-4008-45d1-8290-875d79f37c01 to worker[182] qsize 0
2022-07-25 23:57:47,094 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2ea33564-f03a-429d-b360-db61777478c6, queue=awx_private_queue)
[2022-07-25 23:57:47,094: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2ea33564-f03a-429d-b360-db61777478c6, queue=awx_private_queue)
2022-07-25 23:57:47,355 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e3e14665-2691-4a5f-a326-a29a9f88afa8, queue=awx_private_queue)
[2022-07-25 23:57:47,355: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e3e14665-2691-4a5f-a326-a29a9f88afa8, queue=awx_private_queue)
2022-07-25 23:57:47,113 DEBUG awx.main.dispatch delivered 2ea33564-f03a-429d-b360-db61777478c6 to worker[181] qsize 0
2022-07-25 23:57:47,388 DEBUG awx.main.dispatch delivered e3e14665-2691-4a5f-a326-a29a9f88afa8 to worker[180] qsize 0
2022-07-25 23:57:47,388 DEBUG awx.main.dispatch task 2ea33564-f03a-429d-b360-db61777478c6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:57:47,390 DEBUG awx.main.dispatch task e3e14665-2691-4a5f-a326-a29a9f88afa8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:57:47,394 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:57:47,006 DEBUG awx.main.dispatch task 215ab6a4-4008-45d1-8290-875d79f37c01 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:57:47,825 DEBUG awx.main.dispatch task 215ab6a4-4008-45d1-8290-875d79f37c01 is finished
2022-07-25 23:57:47,885 DEBUG awx.main.dispatch task 686ef635-5dd6-4476-8cd5-f981924ae9fa is finished
2022-07-25 23:57:48,098 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:57:48,158 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:57:17.166527+00:00
2022-07-25 23:57:48,284 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:57:48,709 DEBUG awx.main.dispatch task 2ea33564-f03a-429d-b360-db61777478c6 is finished
2022-07-25 23:57:49,241 DEBUG awx.main.dispatch task e3e14665-2691-4a5f-a326-a29a9f88afa8 is finished
RESULT 2
OKREADY
2022-07-25 23:58:07,365 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bab9c907-d52e-4d8a-8800-89fe232a999e, queue=awx_private_queue)
[2022-07-25 23:58:07,365: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bab9c907-d52e-4d8a-8800-89fe232a999e, queue=awx_private_queue)
2022-07-25 23:58:07,384 DEBUG awx.main.dispatch delivered bab9c907-d52e-4d8a-8800-89fe232a999e to worker[183] qsize 0
2022-07-25 23:58:07,386 DEBUG awx.main.dispatch task bab9c907-d52e-4d8a-8800-89fe232a999e starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:58:07,388 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:58:07,522 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:58:07,731 DEBUG awx.main.dispatch task bab9c907-d52e-4d8a-8800-89fe232a999e is finished
2022-07-25 23:58:17,094 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ef8c18a1-ecfb-489b-902b-c3e36cf7d7c5, queue=awx_private_queue)
[2022-07-25 23:58:17,094: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ef8c18a1-ecfb-489b-902b-c3e36cf7d7c5, queue=awx_private_queue)
2022-07-25 23:58:17,114 DEBUG awx.main.dispatch delivered ef8c18a1-ecfb-489b-902b-c3e36cf7d7c5 to worker[183] qsize 0
2022-07-25 23:58:17,117 DEBUG awx.main.dispatch task ef8c18a1-ecfb-489b-902b-c3e36cf7d7c5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:58:17,177 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:58:17,185 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:57:48.098512+00:00
2022-07-25 23:58:17,304 DEBUG awx.main.dispatch task ef8c18a1-ecfb-489b-902b-c3e36cf7d7c5 is finished
2022-07-25 23:58:27,366 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(34c7c65d-c10b-4828-9764-e5e06f7af78e, queue=awx_private_queue)
[2022-07-25 23:58:27,366: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(34c7c65d-c10b-4828-9764-e5e06f7af78e, queue=awx_private_queue)
2022-07-25 23:58:27,386 DEBUG awx.main.dispatch delivered 34c7c65d-c10b-4828-9764-e5e06f7af78e to worker[181] qsize 0
2022-07-25 23:58:27,390 DEBUG awx.main.dispatch task 34c7c65d-c10b-4828-9764-e5e06f7af78e starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:58:27,392 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:58:27,620 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:58:27,964 DEBUG awx.main.dispatch task 34c7c65d-c10b-4828-9764-e5e06f7af78e is finished
2022-07-25 23:58:46,674 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(19e49438-6013-40fc-a998-6adec12ff47c, queue=onap-awx-0)
[2022-07-25 23:58:46,674: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(19e49438-6013-40fc-a998-6adec12ff47c, queue=onap-awx-0)
2022-07-25 23:58:46,915 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(6cffb8ea-000c-4978-9b2a-47d60f62deed, queue=onap-awx-0)
[2022-07-25 23:58:46,915: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(6cffb8ea-000c-4978-9b2a-47d60f62deed, queue=onap-awx-0)
2022-07-25 23:58:47,025 DEBUG awx.main.dispatch delivered 19e49438-6013-40fc-a998-6adec12ff47c to worker[181] qsize 0
2022-07-25 23:58:47,029 DEBUG awx.main.dispatch delivered 6cffb8ea-000c-4978-9b2a-47d60f62deed to worker[183] qsize 0
2022-07-25 23:58:47,029 DEBUG awx.main.dispatch task 19e49438-6013-40fc-a998-6adec12ff47c starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:58:47,030 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:58:47,032 DEBUG awx.main.dispatch task 6cffb8ea-000c-4978-9b2a-47d60f62deed starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:58:47,095 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(656697c8-5ba5-42f2-ba93-92cd103831dc, queue=awx_private_queue)
[2022-07-25 23:58:47,095: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(656697c8-5ba5-42f2-ba93-92cd103831dc, queue=awx_private_queue)
2022-07-25 23:58:47,111 DEBUG awx.main.dispatch delivered 656697c8-5ba5-42f2-ba93-92cd103831dc to worker[180] qsize 0
2022-07-25 23:58:47,114 DEBUG awx.main.dispatch task 656697c8-5ba5-42f2-ba93-92cd103831dc starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:58:47,168 DEBUG awx.main.dispatch task 6cffb8ea-000c-4978-9b2a-47d60f62deed is finished
2022-07-25 23:58:47,218 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:58:47,229 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:58:17.177832+00:00
2022-07-25 23:58:47,272 DEBUG awx.main.dispatch task 19e49438-6013-40fc-a998-6adec12ff47c is finished
2022-07-25 23:58:47,332 DEBUG awx.main.dispatch task 656697c8-5ba5-42f2-ba93-92cd103831dc is finished
2022-07-25 23:58:47,366 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d97370f2-840d-48ad-a57b-84a2fef6d20a, queue=awx_private_queue)
[2022-07-25 23:58:47,366: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d97370f2-840d-48ad-a57b-84a2fef6d20a, queue=awx_private_queue)
2022-07-25 23:58:47,384 DEBUG awx.main.dispatch delivered d97370f2-840d-48ad-a57b-84a2fef6d20a to worker[182] qsize 0
2022-07-25 23:58:47,613 DEBUG awx.main.dispatch task d97370f2-840d-48ad-a57b-84a2fef6d20a starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:58:47,616 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:58:47,901 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:58:48,179 DEBUG awx.main.dispatch task d97370f2-840d-48ad-a57b-84a2fef6d20a is finished
RESULT 2
OKREADY
2022-07-25 23:59:07,371 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(42acaa5b-f681-4e4b-81e8-95ffa7e5ec2a, queue=awx_private_queue)
[2022-07-25 23:59:07,371: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(42acaa5b-f681-4e4b-81e8-95ffa7e5ec2a, queue=awx_private_queue)
2022-07-25 23:59:07,395 DEBUG awx.main.dispatch delivered 42acaa5b-f681-4e4b-81e8-95ffa7e5ec2a to worker[181] qsize 0
2022-07-25 23:59:07,398 DEBUG awx.main.dispatch task 42acaa5b-f681-4e4b-81e8-95ffa7e5ec2a starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:59:07,401 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:59:07,488 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:59:07,740 DEBUG awx.main.dispatch task 42acaa5b-f681-4e4b-81e8-95ffa7e5ec2a is finished
2022-07-25 23:59:17,095 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e003fa30-323d-416b-8c75-17ef049547b2, queue=awx_private_queue)
[2022-07-25 23:59:17,095: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e003fa30-323d-416b-8c75-17ef049547b2, queue=awx_private_queue)
2022-07-25 23:59:17,113 DEBUG awx.main.dispatch delivered e003fa30-323d-416b-8c75-17ef049547b2 to worker[181] qsize 0
2022-07-25 23:59:17,116 DEBUG awx.main.dispatch task e003fa30-323d-416b-8c75-17ef049547b2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:59:17,171 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:59:17,179 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:58:47.219386+00:00
2022-07-25 23:59:17,301 DEBUG awx.main.dispatch task e003fa30-323d-416b-8c75-17ef049547b2 is finished
2022-07-25 23:59:27,371 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1fee2dbc-e73b-474d-a289-d65c3eca0646, queue=awx_private_queue)
[2022-07-25 23:59:27,371: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1fee2dbc-e73b-474d-a289-d65c3eca0646, queue=awx_private_queue)
2022-07-25 23:59:27,390 DEBUG awx.main.dispatch delivered 1fee2dbc-e73b-474d-a289-d65c3eca0646 to worker[181] qsize 0
2022-07-25 23:59:27,393 DEBUG awx.main.dispatch task 1fee2dbc-e73b-474d-a289-d65c3eca0646 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:59:27,394 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:59:27,620 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:59:28,023 DEBUG awx.main.dispatch task 1fee2dbc-e73b-474d-a289-d65c3eca0646 is finished
2022-07-25 23:59:46,678 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(dc8f586f-01d4-4fd6-92bb-037893e4d734, queue=onap-awx-0)
[2022-07-25 23:59:46,678: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(dc8f586f-01d4-4fd6-92bb-037893e4d734, queue=onap-awx-0)
2022-07-25 23:59:46,860 DEBUG awx.main.dispatch delivered dc8f586f-01d4-4fd6-92bb-037893e4d734 to worker[180] qsize 0
2022-07-25 23:59:46,915 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(106f2ed9-d68f-4bbb-94b3-ec5fc89615bd, queue=onap-awx-0)
[2022-07-25 23:59:46,915: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(106f2ed9-d68f-4bbb-94b3-ec5fc89615bd, queue=onap-awx-0)
2022-07-25 23:59:46,932 DEBUG awx.main.dispatch delivered 106f2ed9-d68f-4bbb-94b3-ec5fc89615bd to worker[183] qsize 0
2022-07-25 23:59:47,095 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(00cf9ffb-fb84-4ae5-ac9b-1e05500304d4, queue=awx_private_queue)
[2022-07-25 23:59:47,095: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(00cf9ffb-fb84-4ae5-ac9b-1e05500304d4, queue=awx_private_queue)
2022-07-25 23:59:47,111 DEBUG awx.main.dispatch delivered 00cf9ffb-fb84-4ae5-ac9b-1e05500304d4 to worker[182] qsize 0
2022-07-25 23:59:47,114 DEBUG awx.main.dispatch task 00cf9ffb-fb84-4ae5-ac9b-1e05500304d4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-25 23:59:47,159 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-25 23:59:47,177 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:59:17.172023+00:00
2022-07-25 23:59:47,290 DEBUG awx.main.dispatch task 00cf9ffb-fb84-4ae5-ac9b-1e05500304d4 is finished
2022-07-25 23:59:46,910 DEBUG awx.main.dispatch task dc8f586f-01d4-4fd6-92bb-037893e4d734 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-25 23:59:47,307 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-25 23:59:46,935 DEBUG awx.main.dispatch task 106f2ed9-d68f-4bbb-94b3-ec5fc89615bd starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-25 23:59:47,371 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d4ca1901-27a3-43fd-800d-609a46f9aa7f, queue=awx_private_queue)
[2022-07-25 23:59:47,371: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d4ca1901-27a3-43fd-800d-609a46f9aa7f, queue=awx_private_queue)
2022-07-25 23:59:47,390 DEBUG awx.main.dispatch delivered d4ca1901-27a3-43fd-800d-609a46f9aa7f to worker[182] qsize 0
2022-07-25 23:59:47,393 DEBUG awx.main.dispatch task d4ca1901-27a3-43fd-800d-609a46f9aa7f starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-25 23:59:47,395 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-25 23:59:47,433 DEBUG awx.main.dispatch task 106f2ed9-d68f-4bbb-94b3-ec5fc89615bd is finished
2022-07-25 23:59:47,451 DEBUG awx.main.dispatch task dc8f586f-01d4-4fd6-92bb-037893e4d734 is finished
2022-07-25 23:59:47,624 DEBUG awx.main.scheduler Starting Scheduler
2022-07-25 23:59:47,991 DEBUG awx.main.dispatch task d4ca1901-27a3-43fd-800d-609a46f9aa7f is finished
2022-07-26 00:00:00,003 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(ab1cbf41-8021-4e91-9cb2-acc404d5bd43, queue=awx_private_queue)
[2022-07-26 00:00:00,003: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(ab1cbf41-8021-4e91-9cb2-acc404d5bd43, queue=awx_private_queue)
RESULT 2
OKREADY
2022-07-26 00:00:00,021 DEBUG awx.main.dispatch delivered ab1cbf41-8021-4e91-9cb2-acc404d5bd43 to worker[183] qsize 0
2022-07-26 00:00:00,024 DEBUG awx.main.dispatch task ab1cbf41-8021-4e91-9cb2-acc404d5bd43 starting awx.main.tasks.gather_analytics(*[])
2022-07-26 00:00:00,026 DEBUG awx.main.dispatch task ab1cbf41-8021-4e91-9cb2-acc404d5bd43 is finished
2022-07-26 00:00:07,370 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b414f979-643b-4966-b595-b24076fa7175, queue=awx_private_queue)
[2022-07-26 00:00:07,370: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b414f979-643b-4966-b595-b24076fa7175, queue=awx_private_queue)
2022-07-26 00:00:07,388 DEBUG awx.main.dispatch delivered b414f979-643b-4966-b595-b24076fa7175 to worker[183] qsize 0
2022-07-26 00:00:07,391 DEBUG awx.main.dispatch task b414f979-643b-4966-b595-b24076fa7175 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:00:07,393 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:00:07,397 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-26 00:00:07,400 DEBUG awx.main.dispatch task b414f979-643b-4966-b595-b24076fa7175 is finished
2022-07-26 00:00:17,095 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(364eb99a-8b8a-4008-b64d-888221e7a52a, queue=awx_private_queue)
[2022-07-26 00:00:17,095: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(364eb99a-8b8a-4008-b64d-888221e7a52a, queue=awx_private_queue)
2022-07-26 00:00:17,114 DEBUG awx.main.dispatch delivered 364eb99a-8b8a-4008-b64d-888221e7a52a to worker[183] qsize 0
2022-07-26 00:00:17,116 DEBUG awx.main.dispatch task 364eb99a-8b8a-4008-b64d-888221e7a52a starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-26 00:00:17,122 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-26 00:00:17,125 DEBUG awx.main.dispatch task 364eb99a-8b8a-4008-b64d-888221e7a52a is finished
2022-07-26 00:00:27,371 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d0efa780-ade6-475a-b62a-b18f45d45ec4, queue=awx_private_queue)
[2022-07-26 00:00:27,371: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d0efa780-ade6-475a-b62a-b18f45d45ec4, queue=awx_private_queue)
2022-07-26 00:00:27,389 DEBUG awx.main.dispatch delivered d0efa780-ade6-475a-b62a-b18f45d45ec4 to worker[182] qsize 0
2022-07-26 00:00:27,392 DEBUG awx.main.dispatch task d0efa780-ade6-475a-b62a-b18f45d45ec4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:00:27,395 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:00:27,398 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-26 00:00:27,401 DEBUG awx.main.dispatch task d0efa780-ade6-475a-b62a-b18f45d45ec4 is finished
2022-07-26 00:00:46,678 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(40ff97d7-8cde-49aa-8ecb-86972416220b, queue=onap-awx-0)
[2022-07-26 00:00:46,678: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(40ff97d7-8cde-49aa-8ecb-86972416220b, queue=onap-awx-0)
2022-07-26 00:00:46,815 DEBUG awx.main.dispatch delivered 40ff97d7-8cde-49aa-8ecb-86972416220b to worker[181] qsize 0
2022-07-26 00:00:46,827 DEBUG awx.main.dispatch task 40ff97d7-8cde-49aa-8ecb-86972416220b starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-26 00:00:46,838 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-26 00:00:46,842 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-26 00:00:46,845 DEBUG awx.main.dispatch task 40ff97d7-8cde-49aa-8ecb-86972416220b is finished
2022-07-26 00:00:46,916 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(068a8d4e-b275-4aec-90d0-81765ac94663, queue=onap-awx-0)
[2022-07-26 00:00:46,916: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(068a8d4e-b275-4aec-90d0-81765ac94663, queue=onap-awx-0)
2022-07-26 00:00:46,934 DEBUG awx.main.dispatch delivered 068a8d4e-b275-4aec-90d0-81765ac94663 to worker[180] qsize 0
2022-07-26 00:00:46,938 DEBUG awx.main.dispatch task 068a8d4e-b275-4aec-90d0-81765ac94663 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-26 00:00:46,956 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-26 00:00:46,959 DEBUG awx.main.dispatch task 068a8d4e-b275-4aec-90d0-81765ac94663 is finished
2022-07-26 00:00:47,095 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6571b8f2-d86d-4b7e-90d9-25dff39c3b00, queue=awx_private_queue)
[2022-07-26 00:00:47,095: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6571b8f2-d86d-4b7e-90d9-25dff39c3b00, queue=awx_private_queue)
2022-07-26 00:00:47,112 DEBUG awx.main.dispatch delivered 6571b8f2-d86d-4b7e-90d9-25dff39c3b00 to worker[180] qsize 0
2022-07-26 00:00:47,115 DEBUG awx.main.dispatch task 6571b8f2-d86d-4b7e-90d9-25dff39c3b00 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-26 00:00:47,118 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-26 00:00:47,120 DEBUG awx.main.dispatch task 6571b8f2-d86d-4b7e-90d9-25dff39c3b00 is finished
2022-07-26 00:00:47,372 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d39efe33-04cc-4a74-b5a7-27c2f75257f5, queue=awx_private_queue)
[2022-07-26 00:00:47,372: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d39efe33-04cc-4a74-b5a7-27c2f75257f5, queue=awx_private_queue)
2022-07-26 00:00:47,402 DEBUG awx.main.dispatch delivered d39efe33-04cc-4a74-b5a7-27c2f75257f5 to worker[183] qsize 0
2022-07-26 00:00:47,645 DEBUG awx.main.dispatch task d39efe33-04cc-4a74-b5a7-27c2f75257f5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:00:47,648 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:00:47,654 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-26 00:00:47,658 DEBUG awx.main.dispatch task d39efe33-04cc-4a74-b5a7-27c2f75257f5 is finished
2022-07-26 00:01:00,004 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(5b66e850-1215-482e-808d-f2506770fe17, queue=awx_private_queue)
[2022-07-26 00:01:00,004: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(5b66e850-1215-482e-808d-f2506770fe17, queue=awx_private_queue)
RESULT 2
OKREADY
2022-07-26 00:01:00,029 DEBUG awx.main.dispatch delivered 5b66e850-1215-482e-808d-f2506770fe17 to worker[180] qsize 0
2022-07-26 00:01:00,033 DEBUG awx.main.dispatch task 5b66e850-1215-482e-808d-f2506770fe17 starting awx.main.tasks.gather_analytics(*[])
2022-07-26 00:01:00,037 DEBUG awx.main.dispatch task 5b66e850-1215-482e-808d-f2506770fe17 is finished
2022-07-26 00:01:07,371 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9a362a8d-83ce-4b28-b2ec-f344daaa089c, queue=awx_private_queue)
[2022-07-26 00:01:07,371: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9a362a8d-83ce-4b28-b2ec-f344daaa089c, queue=awx_private_queue)
2022-07-26 00:01:07,398 DEBUG awx.main.dispatch delivered 9a362a8d-83ce-4b28-b2ec-f344daaa089c to worker[183] qsize 0
2022-07-26 00:01:07,400 DEBUG awx.main.dispatch task 9a362a8d-83ce-4b28-b2ec-f344daaa089c starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:01:07,402 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:01:07,508 DEBUG awx.main.scheduler Starting Scheduler
2022-07-26 00:01:07,657 DEBUG awx.main.dispatch task 9a362a8d-83ce-4b28-b2ec-f344daaa089c is finished
2022-07-26 00:01:17,096 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3bac4901-70dc-44a9-9a74-d468142abb0c, queue=awx_private_queue)
[2022-07-26 00:01:17,096: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3bac4901-70dc-44a9-9a74-d468142abb0c, queue=awx_private_queue)
2022-07-26 00:01:17,118 DEBUG awx.main.dispatch delivered 3bac4901-70dc-44a9-9a74-d468142abb0c to worker[180] qsize 0
2022-07-26 00:01:17,120 DEBUG awx.main.dispatch task 3bac4901-70dc-44a9-9a74-d468142abb0c starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-26 00:01:17,234 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-26 00:01:17,270 DEBUG awx.main.tasks Last scheduler run was: 2022-07-25 23:59:47.160273+00:00
2022-07-26 00:01:17,404 DEBUG awx.main.dispatch task 3bac4901-70dc-44a9-9a74-d468142abb0c is finished
2022-07-26 00:01:27,373 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fc5d6a8f-71df-4dcb-9ad3-a8d4732a65d4, queue=awx_private_queue)
[2022-07-26 00:01:27,373: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fc5d6a8f-71df-4dcb-9ad3-a8d4732a65d4, queue=awx_private_queue)
2022-07-26 00:01:27,393 DEBUG awx.main.dispatch delivered fc5d6a8f-71df-4dcb-9ad3-a8d4732a65d4 to worker[183] qsize 0
2022-07-26 00:01:27,397 DEBUG awx.main.dispatch task fc5d6a8f-71df-4dcb-9ad3-a8d4732a65d4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:01:27,399 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:01:27,532 DEBUG awx.main.scheduler Starting Scheduler
2022-07-26 00:01:27,854 DEBUG awx.main.dispatch task fc5d6a8f-71df-4dcb-9ad3-a8d4732a65d4 is finished
2022-07-26 00:01:46,684 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f3b15da1-e9a5-4bae-a9f1-e14868118d75, queue=onap-awx-0)
[2022-07-26 00:01:46,684: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f3b15da1-e9a5-4bae-a9f1-e14868118d75, queue=onap-awx-0)
2022-07-26 00:01:46,843 DEBUG awx.main.dispatch delivered f3b15da1-e9a5-4bae-a9f1-e14868118d75 to worker[181] qsize 0
2022-07-26 00:01:46,916 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(6b2b6a31-0091-48bb-a566-b13364cc4ffd, queue=onap-awx-0)
[2022-07-26 00:01:46,916: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(6b2b6a31-0091-48bb-a566-b13364cc4ffd, queue=onap-awx-0)
2022-07-26 00:01:46,932 DEBUG awx.main.dispatch delivered 6b2b6a31-0091-48bb-a566-b13364cc4ffd to worker[182] qsize 0
2022-07-26 00:01:47,095 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ef0641a1-a372-4d1e-9da0-9910527719f0, queue=awx_private_queue)
[2022-07-26 00:01:47,095: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ef0641a1-a372-4d1e-9da0-9910527719f0, queue=awx_private_queue)
2022-07-26 00:01:47,113 DEBUG awx.main.dispatch delivered ef0641a1-a372-4d1e-9da0-9910527719f0 to worker[180] qsize 0
2022-07-26 00:01:46,871 DEBUG awx.main.dispatch task f3b15da1-e9a5-4bae-a9f1-e14868118d75 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-26 00:01:47,215 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-26 00:01:47,333 DEBUG awx.main.dispatch task f3b15da1-e9a5-4bae-a9f1-e14868118d75 is finished
2022-07-26 00:01:46,935 DEBUG awx.main.dispatch task 6b2b6a31-0091-48bb-a566-b13364cc4ffd starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-26 00:01:47,371 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5c27ec24-90ba-4e78-b816-d4caf46e1ccd, queue=awx_private_queue)
[2022-07-26 00:01:47,371: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5c27ec24-90ba-4e78-b816-d4caf46e1ccd, queue=awx_private_queue)
2022-07-26 00:01:47,390 DEBUG awx.main.dispatch delivered 5c27ec24-90ba-4e78-b816-d4caf46e1ccd to worker[183] qsize 0
2022-07-26 00:01:47,393 DEBUG awx.main.dispatch task 5c27ec24-90ba-4e78-b816-d4caf46e1ccd starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:01:47,394 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:01:47,115 DEBUG awx.main.dispatch task ef0641a1-a372-4d1e-9da0-9910527719f0 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-26 00:01:47,446 DEBUG awx.main.dispatch task 6b2b6a31-0091-48bb-a566-b13364cc4ffd is finished
2022-07-26 00:01:47,466 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-26 00:01:47,490 DEBUG awx.main.tasks Last scheduler run was: 2022-07-26 00:01:17.235304+00:00
2022-07-26 00:01:47,550 DEBUG awx.main.scheduler Starting Scheduler
2022-07-26 00:01:47,653 DEBUG awx.main.dispatch task ef0641a1-a372-4d1e-9da0-9910527719f0 is finished
2022-07-26 00:01:47,734 DEBUG awx.main.dispatch task 5c27ec24-90ba-4e78-b816-d4caf46e1ccd is finished
2022-07-26 00:02:00,003 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(5eca647a-9e3d-4b7e-9808-6b076273316e, queue=awx_private_queue)
[2022-07-26 00:02:00,003: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(5eca647a-9e3d-4b7e-9808-6b076273316e, queue=awx_private_queue)
2022-07-26 00:02:00,021 DEBUG awx.main.dispatch delivered 5eca647a-9e3d-4b7e-9808-6b076273316e to worker[183] qsize 0
RESULT 2
OKREADY
2022-07-26 00:02:00,027 DEBUG awx.main.dispatch task 5eca647a-9e3d-4b7e-9808-6b076273316e starting awx.main.tasks.gather_analytics(*[])
2022-07-26 00:02:00,030 DEBUG awx.main.dispatch task 5eca647a-9e3d-4b7e-9808-6b076273316e is finished
2022-07-26 00:02:07,371 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f12c59c2-5b35-4425-829d-8d1bde32fa50, queue=awx_private_queue)
[2022-07-26 00:02:07,371: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f12c59c2-5b35-4425-829d-8d1bde32fa50, queue=awx_private_queue)
2022-07-26 00:02:07,390 DEBUG awx.main.dispatch delivered f12c59c2-5b35-4425-829d-8d1bde32fa50 to worker[181] qsize 0
2022-07-26 00:02:07,392 DEBUG awx.main.dispatch task f12c59c2-5b35-4425-829d-8d1bde32fa50 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:02:07,395 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:02:07,607 DEBUG awx.main.scheduler Starting Scheduler
2022-07-26 00:02:07,767 DEBUG awx.main.dispatch task f12c59c2-5b35-4425-829d-8d1bde32fa50 is finished
2022-07-26 00:02:17,107 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(def6da83-f322-4f2a-869d-7bb898bc4e2d, queue=awx_private_queue)
[2022-07-26 00:02:17,107: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(def6da83-f322-4f2a-869d-7bb898bc4e2d, queue=awx_private_queue)
2022-07-26 00:02:17,125 DEBUG awx.main.dispatch delivered def6da83-f322-4f2a-869d-7bb898bc4e2d to worker[180] qsize 0
2022-07-26 00:02:17,128 DEBUG awx.main.dispatch task def6da83-f322-4f2a-869d-7bb898bc4e2d starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-26 00:02:17,230 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-26 00:02:17,239 DEBUG awx.main.tasks Last scheduler run was: 2022-07-26 00:01:47.467002+00:00
2022-07-26 00:02:17,400 DEBUG awx.main.dispatch task def6da83-f322-4f2a-869d-7bb898bc4e2d is finished
2022-07-26 00:02:27,371 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0a41ad33-e4f3-4c18-aa1d-76d1118a6344, queue=awx_private_queue)
[2022-07-26 00:02:27,371: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0a41ad33-e4f3-4c18-aa1d-76d1118a6344, queue=awx_private_queue)
2022-07-26 00:02:27,389 DEBUG awx.main.dispatch delivered 0a41ad33-e4f3-4c18-aa1d-76d1118a6344 to worker[182] qsize 0
2022-07-26 00:02:27,391 DEBUG awx.main.dispatch task 0a41ad33-e4f3-4c18-aa1d-76d1118a6344 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:02:27,393 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:02:27,554 DEBUG awx.main.scheduler Starting Scheduler
2022-07-26 00:02:27,842 DEBUG awx.main.dispatch task 0a41ad33-e4f3-4c18-aa1d-76d1118a6344 is finished
2022-07-26 00:02:46,691 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b39145ae-b359-47ac-b95c-8b06f83a052b, queue=onap-awx-0)
[2022-07-26 00:02:46,691: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b39145ae-b359-47ac-b95c-8b06f83a052b, queue=onap-awx-0)
2022-07-26 00:02:46,924 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(aa48c8ad-9611-42a2-881d-b13d8e9ada65, queue=onap-awx-0)
[2022-07-26 00:02:46,924: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(aa48c8ad-9611-42a2-881d-b13d8e9ada65, queue=onap-awx-0)
2022-07-26 00:02:47,032 DEBUG awx.main.dispatch delivered b39145ae-b359-47ac-b95c-8b06f83a052b to worker[183] qsize 0
2022-07-26 00:02:47,034 DEBUG awx.main.dispatch delivered aa48c8ad-9611-42a2-881d-b13d8e9ada65 to worker[180] qsize 0
2022-07-26 00:02:47,034 DEBUG awx.main.dispatch task b39145ae-b359-47ac-b95c-8b06f83a052b starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-26 00:02:47,036 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-26 00:02:47,037 DEBUG awx.main.dispatch task aa48c8ad-9611-42a2-881d-b13d8e9ada65 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-26 00:02:47,107 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f6ba204d-bc7d-4527-a106-1a72505d697c, queue=awx_private_queue)
[2022-07-26 00:02:47,107: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f6ba204d-bc7d-4527-a106-1a72505d697c, queue=awx_private_queue)
2022-07-26 00:02:47,127 DEBUG awx.main.dispatch delivered f6ba204d-bc7d-4527-a106-1a72505d697c to worker[181] qsize 0
2022-07-26 00:02:47,320 DEBUG awx.main.dispatch task b39145ae-b359-47ac-b95c-8b06f83a052b is finished
2022-07-26 00:02:47,371 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5b7e2790-cc00-40b5-ae92-4ff477051d02, queue=awx_private_queue)
[2022-07-26 00:02:47,371: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5b7e2790-cc00-40b5-ae92-4ff477051d02, queue=awx_private_queue)
2022-07-26 00:02:47,390 DEBUG awx.main.dispatch delivered 5b7e2790-cc00-40b5-ae92-4ff477051d02 to worker[183] qsize 0
2022-07-26 00:02:47,393 DEBUG awx.main.dispatch task 5b7e2790-cc00-40b5-ae92-4ff477051d02 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:02:47,395 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:02:47,174 DEBUG awx.main.dispatch task aa48c8ad-9611-42a2-881d-b13d8e9ada65 is finished
2022-07-26 00:02:47,130 DEBUG awx.main.dispatch task f6ba204d-bc7d-4527-a106-1a72505d697c starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-26 00:02:47,438 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-26 00:02:47,462 DEBUG awx.main.tasks Last scheduler run was: 2022-07-26 00:02:17.230735+00:00
2022-07-26 00:02:47,471 DEBUG awx.main.scheduler Starting Scheduler
2022-07-26 00:02:47,630 DEBUG awx.main.dispatch task f6ba204d-bc7d-4527-a106-1a72505d697c is finished
2022-07-26 00:02:47,697 DEBUG awx.main.dispatch task 5b7e2790-cc00-40b5-ae92-4ff477051d02 is finished
2022-07-26 00:03:00,004 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(0c1462f5-5640-4397-9967-a9e99cc4d68d, queue=awx_private_queue)
[2022-07-26 00:03:00,004: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(0c1462f5-5640-4397-9967-a9e99cc4d68d, queue=awx_private_queue)
2022-07-26 00:03:00,023 DEBUG awx.main.dispatch delivered 0c1462f5-5640-4397-9967-a9e99cc4d68d to worker[181] qsize 0
RESULT 2
OKREADY
2022-07-26 00:03:00,027 DEBUG awx.main.dispatch task 0c1462f5-5640-4397-9967-a9e99cc4d68d starting awx.main.tasks.gather_analytics(*[])
2022-07-26 00:03:00,030 DEBUG awx.main.dispatch task 0c1462f5-5640-4397-9967-a9e99cc4d68d is finished
2022-07-26 00:03:07,372 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(29683e77-f1cf-4d6e-8a5e-01fb2f2c533b, queue=awx_private_queue)
[2022-07-26 00:03:07,372: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(29683e77-f1cf-4d6e-8a5e-01fb2f2c533b, queue=awx_private_queue)
2022-07-26 00:03:07,391 DEBUG awx.main.dispatch delivered 29683e77-f1cf-4d6e-8a5e-01fb2f2c533b to worker[181] qsize 0
2022-07-26 00:03:07,394 DEBUG awx.main.dispatch task 29683e77-f1cf-4d6e-8a5e-01fb2f2c533b starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:03:07,396 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:03:07,522 DEBUG awx.main.scheduler Starting Scheduler
2022-07-26 00:03:07,753 DEBUG awx.main.dispatch task 29683e77-f1cf-4d6e-8a5e-01fb2f2c533b is finished
2022-07-26 00:03:17,107 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b8dc17f4-4ba7-42db-98f0-4c50ab5a50d1, queue=awx_private_queue)
[2022-07-26 00:03:17,107: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b8dc17f4-4ba7-42db-98f0-4c50ab5a50d1, queue=awx_private_queue)
2022-07-26 00:03:17,127 DEBUG awx.main.dispatch delivered b8dc17f4-4ba7-42db-98f0-4c50ab5a50d1 to worker[182] qsize 0
2022-07-26 00:03:17,130 DEBUG awx.main.dispatch task b8dc17f4-4ba7-42db-98f0-4c50ab5a50d1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-26 00:03:17,199 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-26 00:03:17,215 DEBUG awx.main.tasks Last scheduler run was: 2022-07-26 00:02:47.439025+00:00
2022-07-26 00:03:17,307 DEBUG awx.main.dispatch task b8dc17f4-4ba7-42db-98f0-4c50ab5a50d1 is finished
2022-07-26 00:03:27,372 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1ea56bc2-d9e7-4310-98c1-ad7e5230a7ce, queue=awx_private_queue)
[2022-07-26 00:03:27,372: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1ea56bc2-d9e7-4310-98c1-ad7e5230a7ce, queue=awx_private_queue)
2022-07-26 00:03:27,399 DEBUG awx.main.dispatch delivered 1ea56bc2-d9e7-4310-98c1-ad7e5230a7ce to worker[180] qsize 0
2022-07-26 00:03:27,402 DEBUG awx.main.dispatch task 1ea56bc2-d9e7-4310-98c1-ad7e5230a7ce starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:03:27,404 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:03:27,698 DEBUG awx.main.scheduler Starting Scheduler
2022-07-26 00:03:28,036 DEBUG awx.main.dispatch task 1ea56bc2-d9e7-4310-98c1-ad7e5230a7ce is finished
2022-07-26 00:03:46,699 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(01a0ef0b-3fa5-4ef1-a811-8002118c2b0d, queue=onap-awx-0)
[2022-07-26 00:03:46,699: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(01a0ef0b-3fa5-4ef1-a811-8002118c2b0d, queue=onap-awx-0)
2022-07-26 00:03:46,785 DEBUG awx.main.dispatch delivered 01a0ef0b-3fa5-4ef1-a811-8002118c2b0d to worker[181] qsize 0
2022-07-26 00:03:46,788 DEBUG awx.main.dispatch task 01a0ef0b-3fa5-4ef1-a811-8002118c2b0d starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-26 00:03:46,790 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-26 00:03:46,918 DEBUG awx.main.dispatch task 01a0ef0b-3fa5-4ef1-a811-8002118c2b0d is finished
2022-07-26 00:03:46,924 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(80f63ecd-0f11-4f74-a249-5edb117deb72, queue=onap-awx-0)
[2022-07-26 00:03:46,924: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(80f63ecd-0f11-4f74-a249-5edb117deb72, queue=onap-awx-0)
2022-07-26 00:03:46,941 DEBUG awx.main.dispatch delivered 80f63ecd-0f11-4f74-a249-5edb117deb72 to worker[182] qsize 0
2022-07-26 00:03:46,956 DEBUG awx.main.dispatch task 80f63ecd-0f11-4f74-a249-5edb117deb72 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-26 00:03:47,076 DEBUG awx.main.dispatch task 80f63ecd-0f11-4f74-a249-5edb117deb72 is finished
2022-07-26 00:03:47,107 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3d65a367-c740-478e-927f-3669573a1240, queue=awx_private_queue)
[2022-07-26 00:03:47,107: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3d65a367-c740-478e-927f-3669573a1240, queue=awx_private_queue)
2022-07-26 00:03:47,123 DEBUG awx.main.dispatch delivered 3d65a367-c740-478e-927f-3669573a1240 to worker[181] qsize 0
2022-07-26 00:03:47,126 DEBUG awx.main.dispatch task 3d65a367-c740-478e-927f-3669573a1240 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-26 00:03:47,185 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-26 00:03:47,190 DEBUG awx.main.tasks Last scheduler run was: 2022-07-26 00:03:17.199889+00:00
2022-07-26 00:03:47,372 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8ec94c8c-84d5-4c34-a040-fecb4da32bf0, queue=awx_private_queue)
[2022-07-26 00:03:47,372: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8ec94c8c-84d5-4c34-a040-fecb4da32bf0, queue=awx_private_queue)
2022-07-26 00:03:47,389 DEBUG awx.main.dispatch delivered 8ec94c8c-84d5-4c34-a040-fecb4da32bf0 to worker[180] qsize 0
2022-07-26 00:03:47,244 DEBUG awx.main.dispatch task 3d65a367-c740-478e-927f-3669573a1240 is finished
2022-07-26 00:03:47,391 DEBUG awx.main.dispatch task 8ec94c8c-84d5-4c34-a040-fecb4da32bf0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:03:47,702 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:03:47,762 DEBUG awx.main.scheduler Starting Scheduler
2022-07-26 00:03:47,974 DEBUG awx.main.dispatch task 8ec94c8c-84d5-4c34-a040-fecb4da32bf0 is finished
2022-07-26 00:04:00,004 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(53ce0e1f-bf7f-4893-9e84-fc158b5b7cad, queue=awx_private_queue)
[2022-07-26 00:04:00,004: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(53ce0e1f-bf7f-4893-9e84-fc158b5b7cad, queue=awx_private_queue)
2022-07-26 00:04:00,022 DEBUG awx.main.dispatch delivered 53ce0e1f-bf7f-4893-9e84-fc158b5b7cad to worker[182] qsize 0
RESULT 2
OKREADY
2022-07-26 00:04:00,026 DEBUG awx.main.dispatch task 53ce0e1f-bf7f-4893-9e84-fc158b5b7cad starting awx.main.tasks.gather_analytics(*[])
2022-07-26 00:04:00,031 DEBUG awx.main.dispatch task 53ce0e1f-bf7f-4893-9e84-fc158b5b7cad is finished
2022-07-26 00:04:07,373 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(34411d2f-a1db-4bf6-9166-fcb90ce6dd6d, queue=awx_private_queue)
[2022-07-26 00:04:07,373: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(34411d2f-a1db-4bf6-9166-fcb90ce6dd6d, queue=awx_private_queue)
2022-07-26 00:04:07,396 DEBUG awx.main.dispatch delivered 34411d2f-a1db-4bf6-9166-fcb90ce6dd6d to worker[183] qsize 0
2022-07-26 00:04:07,401 DEBUG awx.main.dispatch task 34411d2f-a1db-4bf6-9166-fcb90ce6dd6d starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:04:07,406 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:04:07,734 DEBUG awx.main.scheduler Starting Scheduler
2022-07-26 00:04:08,177 DEBUG awx.main.dispatch task 34411d2f-a1db-4bf6-9166-fcb90ce6dd6d is finished
2022-07-26 00:04:17,108 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(def52f23-c1cb-4149-be2d-03b006de944b, queue=awx_private_queue)
[2022-07-26 00:04:17,108: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(def52f23-c1cb-4149-be2d-03b006de944b, queue=awx_private_queue)
2022-07-26 00:04:17,133 DEBUG awx.main.dispatch delivered def52f23-c1cb-4149-be2d-03b006de944b to worker[183] qsize 0
2022-07-26 00:04:17,137 DEBUG awx.main.dispatch task def52f23-c1cb-4149-be2d-03b006de944b starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-26 00:04:17,297 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-26 00:04:17,316 DEBUG awx.main.tasks Last scheduler run was: 2022-07-26 00:03:47.185878+00:00
2022-07-26 00:04:17,471 DEBUG awx.main.dispatch task def52f23-c1cb-4149-be2d-03b006de944b is finished
2022-07-26 00:04:27,372 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fad3826d-30f4-46ff-b8eb-79a69023a7ca, queue=awx_private_queue)
[2022-07-26 00:04:27,372: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fad3826d-30f4-46ff-b8eb-79a69023a7ca, queue=awx_private_queue)
2022-07-26 00:04:27,393 DEBUG awx.main.dispatch delivered fad3826d-30f4-46ff-b8eb-79a69023a7ca to worker[183] qsize 0
2022-07-26 00:04:27,397 DEBUG awx.main.dispatch task fad3826d-30f4-46ff-b8eb-79a69023a7ca starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:04:27,398 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:04:27,581 DEBUG awx.main.scheduler Starting Scheduler
2022-07-26 00:04:27,839 DEBUG awx.main.dispatch task fad3826d-30f4-46ff-b8eb-79a69023a7ca is finished
2022-07-26 00:04:46,706 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(77433a71-6d70-4948-a2d6-c6b9a1cd4d9d, queue=onap-awx-0)
[2022-07-26 00:04:46,706: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(77433a71-6d70-4948-a2d6-c6b9a1cd4d9d, queue=onap-awx-0)
2022-07-26 00:04:46,811 DEBUG awx.main.dispatch delivered 77433a71-6d70-4948-a2d6-c6b9a1cd4d9d to worker[180] qsize 0
2022-07-26 00:04:46,817 DEBUG awx.main.dispatch task 77433a71-6d70-4948-a2d6-c6b9a1cd4d9d starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-26 00:04:46,829 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-26 00:04:46,837 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-26 00:04:46,840 DEBUG awx.main.dispatch task 77433a71-6d70-4948-a2d6-c6b9a1cd4d9d is finished
2022-07-26 00:04:46,924 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(79702b28-7117-47dd-8a66-8550ce19f009, queue=onap-awx-0)
[2022-07-26 00:04:46,924: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(79702b28-7117-47dd-8a66-8550ce19f009, queue=onap-awx-0)
2022-07-26 00:04:46,942 DEBUG awx.main.dispatch delivered 79702b28-7117-47dd-8a66-8550ce19f009 to worker[181] qsize 0
2022-07-26 00:04:46,946 DEBUG awx.main.dispatch task 79702b28-7117-47dd-8a66-8550ce19f009 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-26 00:04:46,953 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-26 00:04:46,955 DEBUG awx.main.dispatch task 79702b28-7117-47dd-8a66-8550ce19f009 is finished
2022-07-26 00:04:47,108 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8a08faa0-90d6-4990-bbb2-51bd79801226, queue=awx_private_queue)
[2022-07-26 00:04:47,108: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8a08faa0-90d6-4990-bbb2-51bd79801226, queue=awx_private_queue)
2022-07-26 00:04:47,131 DEBUG awx.main.dispatch delivered 8a08faa0-90d6-4990-bbb2-51bd79801226 to worker[180] qsize 0
2022-07-26 00:04:47,154 DEBUG awx.main.dispatch task 8a08faa0-90d6-4990-bbb2-51bd79801226 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-26 00:04:47,159 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-26 00:04:47,161 DEBUG awx.main.dispatch task 8a08faa0-90d6-4990-bbb2-51bd79801226 is finished
2022-07-26 00:04:47,372 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(935f30d2-5945-46b7-9c56-e77b386ac083, queue=awx_private_queue)
[2022-07-26 00:04:47,372: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(935f30d2-5945-46b7-9c56-e77b386ac083, queue=awx_private_queue)
2022-07-26 00:04:47,390 DEBUG awx.main.dispatch delivered 935f30d2-5945-46b7-9c56-e77b386ac083 to worker[183] qsize 0
2022-07-26 00:04:47,392 DEBUG awx.main.dispatch task 935f30d2-5945-46b7-9c56-e77b386ac083 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:04:47,394 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:04:47,397 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-26 00:04:47,399 DEBUG awx.main.dispatch task 935f30d2-5945-46b7-9c56-e77b386ac083 is finished
2022-07-26 00:05:00,002 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(b579475b-8dd8-4c76-bf99-37b0371c651f, queue=awx_private_queue)
[2022-07-26 00:05:00,002: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(b579475b-8dd8-4c76-bf99-37b0371c651f, queue=awx_private_queue)
2022-07-26 00:05:00,020 DEBUG awx.main.dispatch delivered b579475b-8dd8-4c76-bf99-37b0371c651f to worker[183] qsize 0
RESULT 2
OKREADY
2022-07-26 00:05:00,023 DEBUG awx.main.dispatch task b579475b-8dd8-4c76-bf99-37b0371c651f starting awx.main.tasks.gather_analytics(*[])
2022-07-26 00:05:00,033 DEBUG awx.main.dispatch task b579475b-8dd8-4c76-bf99-37b0371c651f is finished
2022-07-26 00:05:07,373 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0a73a4b5-f0d0-45f0-975d-6e060227d3eb, queue=awx_private_queue)
[2022-07-26 00:05:07,373: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0a73a4b5-f0d0-45f0-975d-6e060227d3eb, queue=awx_private_queue)
2022-07-26 00:05:07,393 DEBUG awx.main.dispatch delivered 0a73a4b5-f0d0-45f0-975d-6e060227d3eb to worker[182] qsize 0
2022-07-26 00:05:07,397 DEBUG awx.main.dispatch task 0a73a4b5-f0d0-45f0-975d-6e060227d3eb starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:05:07,418 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:05:07,422 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-26 00:05:07,426 DEBUG awx.main.dispatch task 0a73a4b5-f0d0-45f0-975d-6e060227d3eb is finished
2022-07-26 00:05:17,108 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b3c29804-756c-4779-a28b-25794b56b659, queue=awx_private_queue)
[2022-07-26 00:05:17,108: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b3c29804-756c-4779-a28b-25794b56b659, queue=awx_private_queue)
2022-07-26 00:05:17,126 DEBUG awx.main.dispatch delivered b3c29804-756c-4779-a28b-25794b56b659 to worker[181] qsize 0
2022-07-26 00:05:17,356 DEBUG awx.main.dispatch task b3c29804-756c-4779-a28b-25794b56b659 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-26 00:05:17,363 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-07-26 00:05:17,366 DEBUG awx.main.dispatch task b3c29804-756c-4779-a28b-25794b56b659 is finished
2022-07-26 00:05:27,374 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e6ccf267-3717-4935-aba2-33bfe3e62b22, queue=awx_private_queue)
[2022-07-26 00:05:27,374: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e6ccf267-3717-4935-aba2-33bfe3e62b22, queue=awx_private_queue)
2022-07-26 00:05:27,395 DEBUG awx.main.dispatch delivered e6ccf267-3717-4935-aba2-33bfe3e62b22 to worker[180] qsize 0
2022-07-26 00:05:27,398 DEBUG awx.main.dispatch task e6ccf267-3717-4935-aba2-33bfe3e62b22 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:05:27,401 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:05:27,596 DEBUG awx.main.scheduler Starting Scheduler
2022-07-26 00:05:27,837 DEBUG awx.main.dispatch task e6ccf267-3717-4935-aba2-33bfe3e62b22 is finished
2022-07-26 00:05:46,716 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(2db5c82d-8e9a-4e25-a7f4-84e734edb133, queue=onap-awx-0)
[2022-07-26 00:05:46,716: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(2db5c82d-8e9a-4e25-a7f4-84e734edb133, queue=onap-awx-0)
2022-07-26 00:05:46,845 DEBUG awx.main.dispatch delivered 2db5c82d-8e9a-4e25-a7f4-84e734edb133 to worker[182] qsize 0
2022-07-26 00:05:46,849 DEBUG awx.main.dispatch task 2db5c82d-8e9a-4e25-a7f4-84e734edb133 starting awx.main.tasks.cluster_node_heartbeat(*[])
2022-07-26 00:05:46,851 DEBUG awx.main.tasks Cluster node heartbeat task.
2022-07-26 00:05:46,924 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8a5a8eed-7446-4537-a4ee-f47a46496ee5, queue=onap-awx-0)
[2022-07-26 00:05:46,924: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8a5a8eed-7446-4537-a4ee-f47a46496ee5, queue=onap-awx-0)
2022-07-26 00:05:46,942 DEBUG awx.main.dispatch delivered 8a5a8eed-7446-4537-a4ee-f47a46496ee5 to worker[183] qsize 0
2022-07-26 00:05:46,944 DEBUG awx.main.dispatch task 8a5a8eed-7446-4537-a4ee-f47a46496ee5 starting awx.main.tasks.awx_k8s_reaper(*[])
2022-07-26 00:05:47,108 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6237d84e-d072-419b-8b0a-8a181e5f2dc1, queue=awx_private_queue)
[2022-07-26 00:05:47,108: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6237d84e-d072-419b-8b0a-8a181e5f2dc1, queue=awx_private_queue)
2022-07-26 00:05:47,112 DEBUG awx.main.dispatch task 2db5c82d-8e9a-4e25-a7f4-84e734edb133 is finished
2022-07-26 00:05:47,126 DEBUG awx.main.dispatch delivered 6237d84e-d072-419b-8b0a-8a181e5f2dc1 to worker[180] qsize 0
2022-07-26 00:05:47,129 DEBUG awx.main.dispatch task 6237d84e-d072-419b-8b0a-8a181e5f2dc1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2022-07-26 00:05:47,129 DEBUG awx.main.dispatch task 8a5a8eed-7446-4537-a4ee-f47a46496ee5 is finished
2022-07-26 00:05:47,169 DEBUG awx.main.tasks Starting periodic scheduler
2022-07-26 00:05:47,198 DEBUG awx.main.tasks Last scheduler run was: 2022-07-26 00:04:17.297994+00:00
2022-07-26 00:05:47,276 DEBUG awx.main.dispatch task 6237d84e-d072-419b-8b0a-8a181e5f2dc1 is finished
2022-07-26 00:05:47,375 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7b17f003-9e4f-4080-9dd8-10dd4e9f8fe6, queue=awx_private_queue)
[2022-07-26 00:05:47,375: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7b17f003-9e4f-4080-9dd8-10dd4e9f8fe6, queue=awx_private_queue)
2022-07-26 00:05:47,395 DEBUG awx.main.dispatch delivered 7b17f003-9e4f-4080-9dd8-10dd4e9f8fe6 to worker[181] qsize 0
2022-07-26 00:05:47,398 DEBUG awx.main.dispatch task 7b17f003-9e4f-4080-9dd8-10dd4e9f8fe6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2022-07-26 00:05:47,400 DEBUG awx.main.scheduler Running Tower task manager.
2022-07-26 00:05:47,624 DEBUG awx.main.scheduler Starting Scheduler
2022-07-26 00:05:48,049 DEBUG awx.main.dispatch task 7b17f003-9e4f-4080-9dd8-10dd4e9f8fe6 is finished
2022-07-26 00:06:00,004 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(03008347-ad09-4c9a-9b4e-b48d0eaa3254, queue=awx_private_queue)
[2022-07-26 00:06:00,004: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(03008347-ad09-4c9a-9b4e-b48d0eaa3254, queue=awx_private_queue)
2022-07-26 00:06:00,022 DEBUG awx.main.dispatch delivered 03008347-ad09-4c9a-9b4e-b48d0eaa3254 to worker[181] qsize 0
RESULT 2
OKREADY
2022-07-26 00:06:00,024 DEBUG awx.main.dispatch task 03008347-ad09-4c9a-9b4e-b48d0eaa3254 starting awx.main.tasks.gather_analytics(*[])
2022-07-26 00:06:00,027 DEBUG awx.main.dispatch task 03008347-ad09-4c9a-9b4e-b48d0eaa3254 is finished
2022-07-26 00:06:07,374 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7eddc7a7-d3a8-4c57-a5c6-5ef9a11b88d0, queue=awx_private_queue)
[2022-07-26 00:06:07,374: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7eddc7a7-d3a8-4c57-a5c6-5ef9a11b88d0, queue=awx_private_queue)
2022-07-26 00:06:07,394 DEBUG awx.main.dispatch delivered 7eddc7a7-d3a8-4c57-a5c6-5ef9a11b88d0 to worker[180] qsize 0
2022-07-26 00:06:07,397 DEBUG awx.main.dispa