By type
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"elapsed": 0,
"match_groupdict": {},
"match_groups": [],
"path": null,
"port": 5432,
"search_regex": null,
"state": "started"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"db": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
}
Default organization added.
Demo Credential, Inventory, and Job Template added.
(changed: True)
2021-04-08 10:46:44,647 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(706af9d2-473e-4e7c-a988-bb47d5488573, queue=awx_private_queue)
Successfully registered instance dev-awx-0
(changed: True)
2021-04-08 10:46:48,276 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(b24f468a-b3d7-4c54-9a98-b98dfa0da134, queue=awx_private_queue)
2021-04-08 10:46:48,381 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(0d4f8083-3bf7-48dc-9a7a-2559cddd6d77, queue=awx_private_queue)
Creating instance group tower
(changed: True)
2021-04-08 10:46:49,123 INFO RPC interface 'supervisor' initialized
2021-04-08 10:46:49,123 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2021-04-08 10:46:49,123 INFO supervisord started with pid 150
2021-04-08 10:46:50,126 INFO spawned: 'awx-config-watcher' with pid 153
2021-04-08 10:46:50,129 INFO spawned: 'dispatcher' with pid 154
2021-04-08 10:46:50,132 INFO spawned: 'callback-receiver' with pid 155
2021-04-08 10:46:50,134 INFO spawned: 'channels-worker' with pid 156
READY
2021-04-08 10:46:51,252 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-08 10:46:51,252 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-08 10:46:51,252 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-08 10:46:51,252 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-08 10:46:53,176 ERROR celery.beat Removing corrupted schedule file '/var/lib/awx/beat.db': error(11, 'Resource temporarily unavailable')
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 485, in setup_schedule
self._store = self._open_schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 475, in _open_schedule
return self.persistence.open(self.schedule_filename, writeback=True)
File "/usr/lib64/python3.6/shelve.py", line 243, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/usr/lib64/python3.6/shelve.py", line 227, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
File "/usr/lib64/python3.6/dbm/__init__.py", line 94, in open
return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable
2021-04-08 10:46:53,178 WARNING awx.main.dispatch scaling up worker pid:176
2021-04-08 10:46:53,191 WARNING awx.main.dispatch scaling up worker pid:177
2021-04-08 10:46:53,204 WARNING awx.main.dispatch scaling up worker pid:178
2021-04-08 10:46:53,219 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:180
2021-04-08 10:46:53,219 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:180
2021-04-08 10:46:53,218 WARNING awx.main.dispatch scaling up worker pid:179
2021-04-08 10:46:53,227 DEBUG awx.main.tasks Syncing Schedules
2021-04-08 10:46:53,228 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:181
2021-04-08 10:46:53,228 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:181
2021-04-08 10:46:53,237 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:182
2021-04-08 10:46:53,237 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:182
2021-04-08 10:46:53,247 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:183
2021-04-08 10:46:53,247 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:183
2021-04-08 10:46:53,332 - INFO - runworker - Using single-threaded worker.
2021-04-08 10:46:53,333 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer)
2021-04-08 10:46:53,333 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive
2021-04-08 10:46:53,458 DEBUG awx.main.tasks Waited 0.0006797313690185547 seconds to obtain lock name: cluster_policy_lock
2021-04-08 10:46:53,468 DEBUG awx.main.tasks Total non-isolated instances:1 available for policy: 1
2021-04-08 10:46:53,468 DEBUG awx.main.tasks Policy percentage, adding Instances [1] to Group tower
2021-04-08 10:46:53,469 DEBUG awx.main.tasks Adding instances [1] to group tower
2021-04-08 10:46:53,476 DEBUG awx.main.tasks Cluster policy computation finished in 0.01763296127319336 seconds
2021-04-08 10:46:53,477 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 10:46:53,514 DEBUG awx.main.dispatch listening on ['awx_private_queue [direct]', 'dev-awx-0 [direct]', 'tower_broadcast_all_dev-awx-0 [fanout]']
RESULT 2
OKREADY
2021-04-08 10:47:13,378 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5de327d6-84f0-43f9-8813-08410706a836, queue=awx_private_queue)
[2021-04-08 10:47:13,378: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5de327d6-84f0-43f9-8813-08410706a836, queue=awx_private_queue)
2021-04-08 10:47:13,410 DEBUG awx.main.dispatch delivered 5de327d6-84f0-43f9-8813-08410706a836 to worker[176] qsize 0
2021-04-08 10:47:13,425 DEBUG awx.main.dispatch task 5de327d6-84f0-43f9-8813-08410706a836 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:47:13,432 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:47:13,470 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:47:13,518 DEBUG awx.main.dispatch task 5de327d6-84f0-43f9-8813-08410706a836 is finished
2021-04-08 10:47:23,327 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(66120f35-b659-4228-95f9-397206ca6bd4, queue=awx_private_queue)
[2021-04-08 10:47:23,327: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(66120f35-b659-4228-95f9-397206ca6bd4, queue=awx_private_queue)
2021-04-08 10:47:23,343 DEBUG awx.main.dispatch delivered 66120f35-b659-4228-95f9-397206ca6bd4 to worker[177] qsize 0
2021-04-08 10:47:23,348 DEBUG awx.main.dispatch task 66120f35-b659-4228-95f9-397206ca6bd4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:47:23,361 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:47:23,389 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:47:23.369917+00:00
2021-04-08 10:47:23,408 DEBUG awx.main.dispatch task 66120f35-b659-4228-95f9-397206ca6bd4 is finished
2021-04-08 10:47:33,347 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b65db990-25fe-4720-95e7-779e0f17643a, queue=awx_private_queue)
[2021-04-08 10:47:33,347: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b65db990-25fe-4720-95e7-779e0f17643a, queue=awx_private_queue)
2021-04-08 10:47:33,377 DEBUG awx.main.dispatch delivered b65db990-25fe-4720-95e7-779e0f17643a to worker[179] qsize 0
2021-04-08 10:47:33,385 DEBUG awx.main.dispatch task b65db990-25fe-4720-95e7-779e0f17643a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:47:33,391 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:47:33,422 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:47:33,471 DEBUG awx.main.dispatch task b65db990-25fe-4720-95e7-779e0f17643a is finished
2021-04-08 10:47:53,349 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b8b1eee9-c8da-4736-894b-dba778901f02, queue=dev-awx-0)
[2021-04-08 10:47:53,349: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b8b1eee9-c8da-4736-894b-dba778901f02, queue=dev-awx-0)
2021-04-08 10:47:53,604 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(10591a60-101d-46a0-9ada-bcda53aeb555, queue=dev-awx-0)
[2021-04-08 10:47:53,604: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(10591a60-101d-46a0-9ada-bcda53aeb555, queue=dev-awx-0)
2021-04-08 10:47:53,627 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5322d810-8b7b-49cf-8c42-34d034debf0d, queue=awx_private_queue)
[2021-04-08 10:47:53,627: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5322d810-8b7b-49cf-8c42-34d034debf0d, queue=awx_private_queue)
2021-04-08 10:47:53,637 DEBUG awx.main.dispatch delivered b8b1eee9-c8da-4736-894b-dba778901f02 to worker[179] qsize 0
2021-04-08 10:47:53,643 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8a8cb4f7-4cf6-417a-8bcb-5ae5c7b9aca8, queue=awx_private_queue)
[2021-04-08 10:47:53,643: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8a8cb4f7-4cf6-417a-8bcb-5ae5c7b9aca8, queue=awx_private_queue)
2021-04-08 10:47:53,643 DEBUG awx.main.dispatch delivered 10591a60-101d-46a0-9ada-bcda53aeb555 to worker[177] qsize 0
2021-04-08 10:47:53,646 DEBUG awx.main.dispatch delivered 5322d810-8b7b-49cf-8c42-34d034debf0d to worker[176] qsize 0
2021-04-08 10:47:53,644 DEBUG awx.main.dispatch task b8b1eee9-c8da-4736-894b-dba778901f02 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 10:47:53,649 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 10:47:53,648 DEBUG awx.main.dispatch task 10591a60-101d-46a0-9ada-bcda53aeb555 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 10:47:53,650 DEBUG awx.main.dispatch task 5322d810-8b7b-49cf-8c42-34d034debf0d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:47:53,664 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:47:53,672 DEBUG awx.main.dispatch delivered 8a8cb4f7-4cf6-417a-8bcb-5ae5c7b9aca8 to worker[178] qsize 0
2021-04-08 10:47:53,671 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:47:23.361690+00:00
2021-04-08 10:47:53,678 DEBUG awx.main.dispatch task 10591a60-101d-46a0-9ada-bcda53aeb555 is finished
2021-04-08 10:47:53,685 DEBUG awx.main.dispatch task b8b1eee9-c8da-4736-894b-dba778901f02 is finished
2021-04-08 10:47:53,680 DEBUG awx.main.dispatch task 8a8cb4f7-4cf6-417a-8bcb-5ae5c7b9aca8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:47:53,690 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:47:53,695 DEBUG awx.main.dispatch task 5322d810-8b7b-49cf-8c42-34d034debf0d is finished
2021-04-08 10:47:53,721 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:47:53,764 DEBUG awx.main.dispatch task 8a8cb4f7-4cf6-417a-8bcb-5ae5c7b9aca8 is finished
RESULT 2
OKREADY
2021-04-08 10:48:13,655 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(80a52ab8-d162-461c-84cb-73a095cd42b1, queue=awx_private_queue)
[2021-04-08 10:48:13,655: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(80a52ab8-d162-461c-84cb-73a095cd42b1, queue=awx_private_queue)
2021-04-08 10:48:13,680 DEBUG awx.main.dispatch delivered 80a52ab8-d162-461c-84cb-73a095cd42b1 to worker[177] qsize 0
2021-04-08 10:48:13,685 DEBUG awx.main.dispatch task 80a52ab8-d162-461c-84cb-73a095cd42b1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:48:13,687 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:48:13,711 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:48:13,756 DEBUG awx.main.dispatch task 80a52ab8-d162-461c-84cb-73a095cd42b1 is finished
2021-04-08 10:48:23,627 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0b1ff6c7-db76-47d3-8895-ae8053c3308e, queue=awx_private_queue)
[2021-04-08 10:48:23,627: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0b1ff6c7-db76-47d3-8895-ae8053c3308e, queue=awx_private_queue)
2021-04-08 10:48:23,646 DEBUG awx.main.dispatch delivered 0b1ff6c7-db76-47d3-8895-ae8053c3308e to worker[176] qsize 0
2021-04-08 10:48:23,648 DEBUG awx.main.dispatch task 0b1ff6c7-db76-47d3-8895-ae8053c3308e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:48:23,656 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:48:23,659 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:47:53.664633+00:00
2021-04-08 10:48:23,672 DEBUG awx.main.dispatch task 0b1ff6c7-db76-47d3-8895-ae8053c3308e is finished
2021-04-08 10:48:33,653 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6acce142-5e82-4606-a083-fede0440dc42, queue=awx_private_queue)
[2021-04-08 10:48:33,653: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6acce142-5e82-4606-a083-fede0440dc42, queue=awx_private_queue)
2021-04-08 10:48:33,667 DEBUG awx.main.dispatch delivered 6acce142-5e82-4606-a083-fede0440dc42 to worker[177] qsize 0
2021-04-08 10:48:33,669 DEBUG awx.main.dispatch task 6acce142-5e82-4606-a083-fede0440dc42 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:48:33,671 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:48:33,699 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:48:33,728 DEBUG awx.main.dispatch task 6acce142-5e82-4606-a083-fede0440dc42 is finished
2021-04-08 10:48:53,365 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(ee083af3-8ee5-4a70-9c78-ca1888ae642c, queue=dev-awx-0)
[2021-04-08 10:48:53,365: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(ee083af3-8ee5-4a70-9c78-ca1888ae642c, queue=dev-awx-0)
2021-04-08 10:48:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(237b0367-b0a7-48b8-a742-8c33e9a6fde9, queue=dev-awx-0)
[2021-04-08 10:48:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(237b0367-b0a7-48b8-a742-8c33e9a6fde9, queue=dev-awx-0)
2021-04-08 10:48:53,638 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(cd3653e4-c1a0-44c2-87e8-5b8f873beb27, queue=awx_private_queue)
[2021-04-08 10:48:53,638: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(cd3653e4-c1a0-44c2-87e8-5b8f873beb27, queue=awx_private_queue)
2021-04-08 10:48:53,645 DEBUG awx.main.dispatch delivered ee083af3-8ee5-4a70-9c78-ca1888ae642c to worker[179] qsize 0
2021-04-08 10:48:53,657 DEBUG awx.main.dispatch delivered 237b0367-b0a7-48b8-a742-8c33e9a6fde9 to worker[177] qsize 0
2021-04-08 10:48:53,660 DEBUG awx.main.dispatch task 237b0367-b0a7-48b8-a742-8c33e9a6fde9 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 10:48:53,668 DEBUG awx.main.dispatch task ee083af3-8ee5-4a70-9c78-ca1888ae642c starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 10:48:53,676 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 10:48:53,700 DEBUG awx.main.dispatch task 237b0367-b0a7-48b8-a742-8c33e9a6fde9 is finished
2021-04-08 10:48:53,702 DEBUG awx.main.dispatch delivered cd3653e4-c1a0-44c2-87e8-5b8f873beb27 to worker[176] qsize 0
2021-04-08 10:48:53,704 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(89b3a5c1-be27-44ed-8121-324bff398556, queue=awx_private_queue)
[2021-04-08 10:48:53,704: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(89b3a5c1-be27-44ed-8121-324bff398556, queue=awx_private_queue)
2021-04-08 10:48:53,709 DEBUG awx.main.dispatch task ee083af3-8ee5-4a70-9c78-ca1888ae642c is finished
2021-04-08 10:48:53,706 DEBUG awx.main.dispatch task cd3653e4-c1a0-44c2-87e8-5b8f873beb27 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:48:53,720 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:48:53,719 DEBUG awx.main.dispatch delivered 89b3a5c1-be27-44ed-8121-324bff398556 to worker[179] qsize 0
2021-04-08 10:48:53,721 DEBUG awx.main.dispatch task 89b3a5c1-be27-44ed-8121-324bff398556 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:48:53,723 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:48:53,725 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:48:23.657044+00:00
2021-04-08 10:48:53,737 DEBUG awx.main.dispatch task cd3653e4-c1a0-44c2-87e8-5b8f873beb27 is finished
2021-04-08 10:48:53,740 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:48:53,769 DEBUG awx.main.dispatch task 89b3a5c1-be27-44ed-8121-324bff398556 is finished
RESULT 2
OKREADY
2021-04-08 10:49:13,706 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(25b391f0-06c0-4c21-8cb0-f9cf54f1df12, queue=awx_private_queue)
[2021-04-08 10:49:13,706: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(25b391f0-06c0-4c21-8cb0-f9cf54f1df12, queue=awx_private_queue)
2021-04-08 10:49:13,731 DEBUG awx.main.dispatch delivered 25b391f0-06c0-4c21-8cb0-f9cf54f1df12 to worker[177] qsize 0
2021-04-08 10:49:13,734 DEBUG awx.main.dispatch task 25b391f0-06c0-4c21-8cb0-f9cf54f1df12 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:49:13,735 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:49:13,756 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:49:13,795 DEBUG awx.main.dispatch task 25b391f0-06c0-4c21-8cb0-f9cf54f1df12 is finished
2021-04-08 10:49:23,638 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c508e457-218a-40d5-ab45-f9e7b3335da7, queue=awx_private_queue)
[2021-04-08 10:49:23,638: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c508e457-218a-40d5-ab45-f9e7b3335da7, queue=awx_private_queue)
2021-04-08 10:49:23,653 DEBUG awx.main.dispatch delivered c508e457-218a-40d5-ab45-f9e7b3335da7 to worker[178] qsize 0
2021-04-08 10:49:23,656 DEBUG awx.main.dispatch task c508e457-218a-40d5-ab45-f9e7b3335da7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:49:23,673 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:49:23,678 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:48:53.720827+00:00
2021-04-08 10:49:23,700 DEBUG awx.main.dispatch task c508e457-218a-40d5-ab45-f9e7b3335da7 is finished
2021-04-08 10:49:33,704 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3d324d95-a539-43fb-89ea-36e56497b502, queue=awx_private_queue)
[2021-04-08 10:49:33,704: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3d324d95-a539-43fb-89ea-36e56497b502, queue=awx_private_queue)
2021-04-08 10:49:33,727 DEBUG awx.main.dispatch delivered 3d324d95-a539-43fb-89ea-36e56497b502 to worker[179] qsize 0
2021-04-08 10:49:33,730 DEBUG awx.main.dispatch task 3d324d95-a539-43fb-89ea-36e56497b502 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:49:33,731 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:49:33,746 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:49:33,778 DEBUG awx.main.dispatch task 3d324d95-a539-43fb-89ea-36e56497b502 is finished
2021-04-08 10:49:53,366 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(05419aab-194b-489a-aea5-5d315084a033, queue=dev-awx-0)
[2021-04-08 10:49:53,366: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(05419aab-194b-489a-aea5-5d315084a033, queue=dev-awx-0)
2021-04-08 10:49:53,420 DEBUG awx.main.dispatch delivered 05419aab-194b-489a-aea5-5d315084a033 to worker[178] qsize 0
2021-04-08 10:49:53,422 DEBUG awx.main.dispatch task 05419aab-194b-489a-aea5-5d315084a033 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 10:49:53,424 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 10:49:53,459 DEBUG awx.main.dispatch task 05419aab-194b-489a-aea5-5d315084a033 is finished
2021-04-08 10:49:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ac537d57-ae9b-4054-be57-bd28cec66077, queue=dev-awx-0)
[2021-04-08 10:49:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ac537d57-ae9b-4054-be57-bd28cec66077, queue=dev-awx-0)
2021-04-08 10:49:53,819 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(38757a64-31b3-4901-9527-ab3885cfea6a, queue=awx_private_queue)
[2021-04-08 10:49:53,819: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(38757a64-31b3-4901-9527-ab3885cfea6a, queue=awx_private_queue)
2021-04-08 10:49:53,818 DEBUG awx.main.dispatch delivered ac537d57-ae9b-4054-be57-bd28cec66077 to worker[177] qsize 0
2021-04-08 10:49:53,821 DEBUG awx.main.dispatch task ac537d57-ae9b-4054-be57-bd28cec66077 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 10:49:53,835 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b431a7ab-bae0-4be2-8c50-07f39fdd2a45, queue=awx_private_queue)
[2021-04-08 10:49:53,835: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b431a7ab-bae0-4be2-8c50-07f39fdd2a45, queue=awx_private_queue)
2021-04-08 10:49:53,836 DEBUG awx.main.dispatch task ac537d57-ae9b-4054-be57-bd28cec66077 is finished
2021-04-08 10:49:53,835 DEBUG awx.main.dispatch delivered 38757a64-31b3-4901-9527-ab3885cfea6a to worker[176] qsize 0
2021-04-08 10:49:53,842 DEBUG awx.main.dispatch task 38757a64-31b3-4901-9527-ab3885cfea6a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:49:53,851 DEBUG awx.main.dispatch delivered b431a7ab-bae0-4be2-8c50-07f39fdd2a45 to worker[179] qsize 0
2021-04-08 10:49:53,855 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:49:53,854 DEBUG awx.main.dispatch task b431a7ab-bae0-4be2-8c50-07f39fdd2a45 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:49:53,856 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:49:53,859 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:49:23.673514+00:00
2021-04-08 10:49:53,875 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:49:53,874 DEBUG awx.main.dispatch task 38757a64-31b3-4901-9527-ab3885cfea6a is finished
2021-04-08 10:49:53,913 DEBUG awx.main.dispatch task b431a7ab-bae0-4be2-8c50-07f39fdd2a45 is finished
RESULT 2
OKREADY
2021-04-08 10:50:13,845 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5ff5da98-486e-4c81-a835-9abf1d63ea1f, queue=awx_private_queue)
[2021-04-08 10:50:13,845: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5ff5da98-486e-4c81-a835-9abf1d63ea1f, queue=awx_private_queue)
2021-04-08 10:50:13,875 DEBUG awx.main.dispatch delivered 5ff5da98-486e-4c81-a835-9abf1d63ea1f to worker[176] qsize 0
2021-04-08 10:50:13,885 DEBUG awx.main.dispatch task 5ff5da98-486e-4c81-a835-9abf1d63ea1f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:50:13,891 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:50:13,914 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:50:13,947 DEBUG awx.main.dispatch task 5ff5da98-486e-4c81-a835-9abf1d63ea1f is finished
2021-04-08 10:50:23,821 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7558647e-8943-4778-b5ab-649d03056d2e, queue=awx_private_queue)
[2021-04-08 10:50:23,821: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7558647e-8943-4778-b5ab-649d03056d2e, queue=awx_private_queue)
2021-04-08 10:50:23,845 DEBUG awx.main.dispatch delivered 7558647e-8943-4778-b5ab-649d03056d2e to worker[176] qsize 0
2021-04-08 10:50:23,848 DEBUG awx.main.dispatch task 7558647e-8943-4778-b5ab-649d03056d2e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:50:23,858 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:50:23,861 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:49:53.856179+00:00
2021-04-08 10:50:23,899 DEBUG awx.main.dispatch task 7558647e-8943-4778-b5ab-649d03056d2e is finished
2021-04-08 10:50:33,843 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(94d0c83f-cd75-4bc7-8c03-a8a12cd548ab, queue=awx_private_queue)
[2021-04-08 10:50:33,843: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(94d0c83f-cd75-4bc7-8c03-a8a12cd548ab, queue=awx_private_queue)
2021-04-08 10:50:33,868 DEBUG awx.main.dispatch delivered 94d0c83f-cd75-4bc7-8c03-a8a12cd548ab to worker[178] qsize 0
2021-04-08 10:50:33,874 DEBUG awx.main.dispatch task 94d0c83f-cd75-4bc7-8c03-a8a12cd548ab starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:50:33,881 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:50:33,904 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:50:33,933 DEBUG awx.main.dispatch task 94d0c83f-cd75-4bc7-8c03-a8a12cd548ab is finished
2021-04-08 10:50:53,367 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b5a0b46e-345f-477e-914e-824c8d10984d, queue=dev-awx-0)
[2021-04-08 10:50:53,367: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b5a0b46e-345f-477e-914e-824c8d10984d, queue=dev-awx-0)
2021-04-08 10:50:53,417 DEBUG awx.main.dispatch delivered b5a0b46e-345f-477e-914e-824c8d10984d to worker[177] qsize 0
2021-04-08 10:50:53,424 DEBUG awx.main.dispatch task b5a0b46e-345f-477e-914e-824c8d10984d starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 10:50:53,429 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 10:50:53,466 DEBUG awx.main.dispatch task b5a0b46e-345f-477e-914e-824c8d10984d is finished
2021-04-08 10:50:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(83d25395-4ff0-4a4b-9c50-575173c68489, queue=dev-awx-0)
[2021-04-08 10:50:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(83d25395-4ff0-4a4b-9c50-575173c68489, queue=dev-awx-0)
2021-04-08 10:50:53,646 DEBUG awx.main.dispatch delivered 83d25395-4ff0-4a4b-9c50-575173c68489 to worker[178] qsize 0
2021-04-08 10:50:53,650 DEBUG awx.main.dispatch task 83d25395-4ff0-4a4b-9c50-575173c68489 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 10:50:53,664 DEBUG awx.main.dispatch task 83d25395-4ff0-4a4b-9c50-575173c68489 is finished
2021-04-08 10:50:53,820 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ac9b7200-cd98-4cdf-9fad-027557f45913, queue=awx_private_queue)
[2021-04-08 10:50:53,820: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ac9b7200-cd98-4cdf-9fad-027557f45913, queue=awx_private_queue)
2021-04-08 10:50:53,843 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0978dce5-f746-48c3-929b-604e725b9edd, queue=awx_private_queue)
[2021-04-08 10:50:53,843: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0978dce5-f746-48c3-929b-604e725b9edd, queue=awx_private_queue)
2021-04-08 10:50:53,838 DEBUG awx.main.dispatch delivered ac9b7200-cd98-4cdf-9fad-027557f45913 to worker[176] qsize 0
2021-04-08 10:50:54,033 DEBUG awx.main.dispatch delivered 0978dce5-f746-48c3-929b-604e725b9edd to worker[179] qsize 0
2021-04-08 10:50:54,033 DEBUG awx.main.dispatch task ac9b7200-cd98-4cdf-9fad-027557f45913 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:50:54,038 DEBUG awx.main.dispatch task 0978dce5-f746-48c3-929b-604e725b9edd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:50:54,046 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:50:54,059 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:50:54,063 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:50:23.858762+00:00
2021-04-08 10:50:54,074 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:50:54,079 DEBUG awx.main.dispatch task ac9b7200-cd98-4cdf-9fad-027557f45913 is finished
2021-04-08 10:50:54,112 DEBUG awx.main.dispatch task 0978dce5-f746-48c3-929b-604e725b9edd is finished
RESULT 2
OKREADY
2021-04-08 10:51:13,849 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(59ccdefe-9a7d-4e6f-af74-872d7f026ac5, queue=awx_private_queue)
[2021-04-08 10:51:13,849: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(59ccdefe-9a7d-4e6f-af74-872d7f026ac5, queue=awx_private_queue)
2021-04-08 10:51:13,882 DEBUG awx.main.dispatch delivered 59ccdefe-9a7d-4e6f-af74-872d7f026ac5 to worker[178] qsize 0
2021-04-08 10:51:13,885 DEBUG awx.main.dispatch task 59ccdefe-9a7d-4e6f-af74-872d7f026ac5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:51:13,888 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:51:13,910 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:51:13,945 DEBUG awx.main.dispatch task 59ccdefe-9a7d-4e6f-af74-872d7f026ac5 is finished
2021-04-08 10:51:23,826 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(08d98793-a316-41fd-bdae-086388d1b312, queue=awx_private_queue)
[2021-04-08 10:51:23,826: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(08d98793-a316-41fd-bdae-086388d1b312, queue=awx_private_queue)
2021-04-08 10:51:23,855 DEBUG awx.main.dispatch delivered 08d98793-a316-41fd-bdae-086388d1b312 to worker[177] qsize 0
2021-04-08 10:51:23,864 DEBUG awx.main.dispatch task 08d98793-a316-41fd-bdae-086388d1b312 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:51:23,881 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:51:23,885 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:50:54.060465+00:00
2021-04-08 10:51:23,901 DEBUG awx.main.dispatch task 08d98793-a316-41fd-bdae-086388d1b312 is finished
2021-04-08 10:51:33,849 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(73a44655-4748-4b46-9aeb-09660e1aa088, queue=awx_private_queue)
[2021-04-08 10:51:33,849: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(73a44655-4748-4b46-9aeb-09660e1aa088, queue=awx_private_queue)
2021-04-08 10:51:33,868 DEBUG awx.main.dispatch delivered 73a44655-4748-4b46-9aeb-09660e1aa088 to worker[179] qsize 0
2021-04-08 10:51:33,872 DEBUG awx.main.dispatch task 73a44655-4748-4b46-9aeb-09660e1aa088 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:51:33,876 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:51:33,898 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:51:33,928 DEBUG awx.main.dispatch task 73a44655-4748-4b46-9aeb-09660e1aa088 is finished
2021-04-08 10:51:53,373 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d5e777e9-6520-439b-842a-3c5871210217, queue=dev-awx-0)
[2021-04-08 10:51:53,373: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d5e777e9-6520-439b-842a-3c5871210217, queue=dev-awx-0)
2021-04-08 10:51:53,413 DEBUG awx.main.dispatch delivered d5e777e9-6520-439b-842a-3c5871210217 to worker[177] qsize 0
2021-04-08 10:51:53,415 DEBUG awx.main.dispatch task d5e777e9-6520-439b-842a-3c5871210217 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 10:51:53,416 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 10:51:53,442 DEBUG awx.main.dispatch task d5e777e9-6520-439b-842a-3c5871210217 is finished
2021-04-08 10:51:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f2cd9185-1364-4cb6-96d8-8609a92be4f5, queue=dev-awx-0)
[2021-04-08 10:51:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f2cd9185-1364-4cb6-96d8-8609a92be4f5, queue=dev-awx-0)
2021-04-08 10:51:53,634 DEBUG awx.main.dispatch delivered f2cd9185-1364-4cb6-96d8-8609a92be4f5 to worker[176] qsize 0
2021-04-08 10:51:53,639 DEBUG awx.main.dispatch task f2cd9185-1364-4cb6-96d8-8609a92be4f5 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 10:51:53,659 DEBUG awx.main.dispatch task f2cd9185-1364-4cb6-96d8-8609a92be4f5 is finished
2021-04-08 10:51:53,820 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5d4e8a1b-a62d-490a-9a51-6b6394834d4f, queue=awx_private_queue)
[2021-04-08 10:51:53,820: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5d4e8a1b-a62d-490a-9a51-6b6394834d4f, queue=awx_private_queue)
2021-04-08 10:51:53,840 DEBUG awx.main.dispatch delivered 5d4e8a1b-a62d-490a-9a51-6b6394834d4f to worker[179] qsize 0
2021-04-08 10:51:53,845 DEBUG awx.main.dispatch task 5d4e8a1b-a62d-490a-9a51-6b6394834d4f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:51:53,850 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cc07a607-7884-437f-8aeb-0575b728e421, queue=awx_private_queue)
[2021-04-08 10:51:53,850: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cc07a607-7884-437f-8aeb-0575b728e421, queue=awx_private_queue)
2021-04-08 10:51:53,854 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:51:53,857 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:51:23.881689+00:00
2021-04-08 10:51:53,873 DEBUG awx.main.dispatch delivered cc07a607-7884-437f-8aeb-0575b728e421 to worker[177] qsize 0
2021-04-08 10:51:53,880 DEBUG awx.main.dispatch task cc07a607-7884-437f-8aeb-0575b728e421 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:51:53,882 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:51:53,874 DEBUG awx.main.dispatch task 5d4e8a1b-a62d-490a-9a51-6b6394834d4f is finished
2021-04-08 10:51:53,897 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:51:53,928 DEBUG awx.main.dispatch task cc07a607-7884-437f-8aeb-0575b728e421 is finished
RESULT 2
OKREADY
2021-04-08 10:52:13,852 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(eb80b553-506f-4884-b78f-64c667a47a1e, queue=awx_private_queue)
[2021-04-08 10:52:13,852: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(eb80b553-506f-4884-b78f-64c667a47a1e, queue=awx_private_queue)
2021-04-08 10:52:13,886 DEBUG awx.main.dispatch delivered eb80b553-506f-4884-b78f-64c667a47a1e to worker[179] qsize 0
2021-04-08 10:52:13,889 DEBUG awx.main.dispatch task eb80b553-506f-4884-b78f-64c667a47a1e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:52:14,078 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:52:14,086 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:52:14,111 DEBUG awx.main.dispatch task eb80b553-506f-4884-b78f-64c667a47a1e is finished
2021-04-08 10:52:23,820 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0a101c08-6858-4023-86e3-c706812230e5, queue=awx_private_queue)
[2021-04-08 10:52:23,820: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0a101c08-6858-4023-86e3-c706812230e5, queue=awx_private_queue)
2021-04-08 10:52:23,854 DEBUG awx.main.dispatch delivered 0a101c08-6858-4023-86e3-c706812230e5 to worker[176] qsize 0
2021-04-08 10:52:23,856 DEBUG awx.main.dispatch task 0a101c08-6858-4023-86e3-c706812230e5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:52:23,869 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:52:23,872 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:51:53.854300+00:00
2021-04-08 10:52:23,884 DEBUG awx.main.dispatch task 0a101c08-6858-4023-86e3-c706812230e5 is finished
2021-04-08 10:52:33,853 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d761b014-cd23-45d9-b5f7-f29f43881120, queue=awx_private_queue)
[2021-04-08 10:52:33,853: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d761b014-cd23-45d9-b5f7-f29f43881120, queue=awx_private_queue)
2021-04-08 10:52:33,867 DEBUG awx.main.dispatch delivered d761b014-cd23-45d9-b5f7-f29f43881120 to worker[177] qsize 0
2021-04-08 10:52:33,870 DEBUG awx.main.dispatch task d761b014-cd23-45d9-b5f7-f29f43881120 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:52:33,873 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:52:33,889 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:52:33,911 DEBUG awx.main.dispatch task d761b014-cd23-45d9-b5f7-f29f43881120 is finished
2021-04-08 10:52:53,375 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(dafc58cb-6854-47ac-a27c-57e91477d990, queue=dev-awx-0)
[2021-04-08 10:52:53,375: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(dafc58cb-6854-47ac-a27c-57e91477d990, queue=dev-awx-0)
2021-04-08 10:52:53,444 DEBUG awx.main.dispatch delivered dafc58cb-6854-47ac-a27c-57e91477d990 to worker[176] qsize 0
2021-04-08 10:52:53,451 DEBUG awx.main.dispatch task dafc58cb-6854-47ac-a27c-57e91477d990 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 10:52:53,458 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 10:52:53,500 DEBUG awx.main.dispatch task dafc58cb-6854-47ac-a27c-57e91477d990 is finished
2021-04-08 10:52:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2ca284e0-f7c1-41ff-9030-6689f212d31a, queue=dev-awx-0)
[2021-04-08 10:52:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2ca284e0-f7c1-41ff-9030-6689f212d31a, queue=dev-awx-0)
2021-04-08 10:52:53,641 DEBUG awx.main.dispatch delivered 2ca284e0-f7c1-41ff-9030-6689f212d31a to worker[177] qsize 0
2021-04-08 10:52:53,645 DEBUG awx.main.dispatch task 2ca284e0-f7c1-41ff-9030-6689f212d31a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 10:52:53,660 DEBUG awx.main.dispatch task 2ca284e0-f7c1-41ff-9030-6689f212d31a is finished
2021-04-08 10:52:53,820 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b8767674-38d7-4fd1-8be7-a5ce14e86d8b, queue=awx_private_queue)
[2021-04-08 10:52:53,820: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b8767674-38d7-4fd1-8be7-a5ce14e86d8b, queue=awx_private_queue)
2021-04-08 10:52:53,835 DEBUG awx.main.dispatch delivered b8767674-38d7-4fd1-8be7-a5ce14e86d8b to worker[176] qsize 0
2021-04-08 10:52:53,838 DEBUG awx.main.dispatch task b8767674-38d7-4fd1-8be7-a5ce14e86d8b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:52:53,846 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:52:53,849 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:52:23.869260+00:00
2021-04-08 10:52:53,852 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0e7a1b6a-a75a-4bbe-a935-494d6fb52398, queue=awx_private_queue)
[2021-04-08 10:52:53,852: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0e7a1b6a-a75a-4bbe-a935-494d6fb52398, queue=awx_private_queue)
2021-04-08 10:52:53,866 DEBUG awx.main.dispatch task b8767674-38d7-4fd1-8be7-a5ce14e86d8b is finished
2021-04-08 10:52:53,874 DEBUG awx.main.dispatch delivered 0e7a1b6a-a75a-4bbe-a935-494d6fb52398 to worker[178] qsize 0
2021-04-08 10:52:53,878 DEBUG awx.main.dispatch task 0e7a1b6a-a75a-4bbe-a935-494d6fb52398 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:52:53,882 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:52:53,908 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:52:53,946 DEBUG awx.main.dispatch task 0e7a1b6a-a75a-4bbe-a935-494d6fb52398 is finished
RESULT 2
OKREADY
2021-04-08 10:53:13,866 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ab4e49ac-cb8f-45a0-b03a-b57887002303, queue=awx_private_queue)
[2021-04-08 10:53:13,866: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ab4e49ac-cb8f-45a0-b03a-b57887002303, queue=awx_private_queue)
2021-04-08 10:53:13,896 DEBUG awx.main.dispatch delivered ab4e49ac-cb8f-45a0-b03a-b57887002303 to worker[176] qsize 0
2021-04-08 10:53:13,900 DEBUG awx.main.dispatch task ab4e49ac-cb8f-45a0-b03a-b57887002303 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:53:13,901 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:53:13,922 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:53:13,956 DEBUG awx.main.dispatch task ab4e49ac-cb8f-45a0-b03a-b57887002303 is finished
2021-04-08 10:53:23,821 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(21b7ef4f-03fd-45f2-b230-d90ca745251c, queue=awx_private_queue)
[2021-04-08 10:53:23,821: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(21b7ef4f-03fd-45f2-b230-d90ca745251c, queue=awx_private_queue)
2021-04-08 10:53:23,864 DEBUG awx.main.dispatch delivered 21b7ef4f-03fd-45f2-b230-d90ca745251c to worker[179] qsize 0
2021-04-08 10:53:23,868 DEBUG awx.main.dispatch task 21b7ef4f-03fd-45f2-b230-d90ca745251c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:53:24,066 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:53:24,069 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:52:53.846859+00:00
2021-04-08 10:53:24,080 DEBUG awx.main.dispatch task 21b7ef4f-03fd-45f2-b230-d90ca745251c is finished
2021-04-08 10:53:33,866 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9f02bd12-03fc-4b18-be8d-fc497861de1f, queue=awx_private_queue)
[2021-04-08 10:53:33,866: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9f02bd12-03fc-4b18-be8d-fc497861de1f, queue=awx_private_queue)
2021-04-08 10:53:33,887 DEBUG awx.main.dispatch delivered 9f02bd12-03fc-4b18-be8d-fc497861de1f to worker[176] qsize 0
2021-04-08 10:53:33,889 DEBUG awx.main.dispatch task 9f02bd12-03fc-4b18-be8d-fc497861de1f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:53:33,891 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:53:33,908 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:53:33,935 DEBUG awx.main.dispatch task 9f02bd12-03fc-4b18-be8d-fc497861de1f is finished
2021-04-08 10:53:53,387 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(9f0ff6ed-ca93-4b36-a25f-ecd1dcf1362c, queue=dev-awx-0)
[2021-04-08 10:53:53,387: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(9f0ff6ed-ca93-4b36-a25f-ecd1dcf1362c, queue=dev-awx-0)
2021-04-08 10:53:53,429 DEBUG awx.main.dispatch delivered 9f0ff6ed-ca93-4b36-a25f-ecd1dcf1362c to worker[177] qsize 0
2021-04-08 10:53:53,434 DEBUG awx.main.dispatch task 9f0ff6ed-ca93-4b36-a25f-ecd1dcf1362c starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 10:53:53,438 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 10:53:53,468 DEBUG awx.main.dispatch task 9f0ff6ed-ca93-4b36-a25f-ecd1dcf1362c is finished
2021-04-08 10:53:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(119e47e6-070e-4948-821f-8abb54fe6a7f, queue=dev-awx-0)
[2021-04-08 10:53:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(119e47e6-070e-4948-821f-8abb54fe6a7f, queue=dev-awx-0)
2021-04-08 10:53:53,632 DEBUG awx.main.dispatch delivered 119e47e6-070e-4948-821f-8abb54fe6a7f to worker[176] qsize 0
2021-04-08 10:53:53,636 DEBUG awx.main.dispatch task 119e47e6-070e-4948-821f-8abb54fe6a7f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 10:53:53,657 DEBUG awx.main.dispatch task 119e47e6-070e-4948-821f-8abb54fe6a7f is finished
2021-04-08 10:53:53,820 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4bd7be01-b3fd-4f28-aede-442af8cc44f1, queue=awx_private_queue)
[2021-04-08 10:53:53,820: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4bd7be01-b3fd-4f28-aede-442af8cc44f1, queue=awx_private_queue)
2021-04-08 10:53:53,833 DEBUG awx.main.dispatch delivered 4bd7be01-b3fd-4f28-aede-442af8cc44f1 to worker[179] qsize 0
2021-04-08 10:53:53,836 DEBUG awx.main.dispatch task 4bd7be01-b3fd-4f28-aede-442af8cc44f1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:53:53,847 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:53:53,851 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:53:24.067234+00:00
2021-04-08 10:53:53,864 DEBUG awx.main.dispatch task 4bd7be01-b3fd-4f28-aede-442af8cc44f1 is finished
2021-04-08 10:53:53,866 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e0a77d4f-dfe3-4204-9bbf-54666ed8c65f, queue=awx_private_queue)
[2021-04-08 10:53:53,866: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e0a77d4f-dfe3-4204-9bbf-54666ed8c65f, queue=awx_private_queue)
2021-04-08 10:53:53,886 DEBUG awx.main.dispatch delivered e0a77d4f-dfe3-4204-9bbf-54666ed8c65f to worker[176] qsize 0
2021-04-08 10:53:53,888 DEBUG awx.main.dispatch task e0a77d4f-dfe3-4204-9bbf-54666ed8c65f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:53:53,890 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:53:53,906 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:53:53,929 DEBUG awx.main.dispatch task e0a77d4f-dfe3-4204-9bbf-54666ed8c65f is finished
RESULT 2
OKREADY
2021-04-08 10:54:13,870 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fa47cc6a-ae58-4091-aa1c-04d35b8bd9af, queue=awx_private_queue)
[2021-04-08 10:54:13,870: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fa47cc6a-ae58-4091-aa1c-04d35b8bd9af, queue=awx_private_queue)
2021-04-08 10:54:13,902 DEBUG awx.main.dispatch delivered fa47cc6a-ae58-4091-aa1c-04d35b8bd9af to worker[176] qsize 0
2021-04-08 10:54:13,905 DEBUG awx.main.dispatch task fa47cc6a-ae58-4091-aa1c-04d35b8bd9af starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:54:13,908 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:54:13,926 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:54:13,958 DEBUG awx.main.dispatch task fa47cc6a-ae58-4091-aa1c-04d35b8bd9af is finished
2021-04-08 10:54:23,820 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a2633b7f-a3f3-4717-9281-89b1ffa43b38, queue=awx_private_queue)
[2021-04-08 10:54:23,820: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a2633b7f-a3f3-4717-9281-89b1ffa43b38, queue=awx_private_queue)
2021-04-08 10:54:23,858 DEBUG awx.main.dispatch delivered a2633b7f-a3f3-4717-9281-89b1ffa43b38 to worker[179] qsize 0
2021-04-08 10:54:23,860 DEBUG awx.main.dispatch task a2633b7f-a3f3-4717-9281-89b1ffa43b38 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:54:23,871 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:54:23,873 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:53:53.847482+00:00
2021-04-08 10:54:23,884 DEBUG awx.main.dispatch task a2633b7f-a3f3-4717-9281-89b1ffa43b38 is finished
2021-04-08 10:54:33,871 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c13298ee-ec2b-43b7-b403-a1b9943af0e8, queue=awx_private_queue)
[2021-04-08 10:54:33,871: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c13298ee-ec2b-43b7-b403-a1b9943af0e8, queue=awx_private_queue)
2021-04-08 10:54:33,886 DEBUG awx.main.dispatch delivered c13298ee-ec2b-43b7-b403-a1b9943af0e8 to worker[176] qsize 0
2021-04-08 10:54:33,888 DEBUG awx.main.dispatch task c13298ee-ec2b-43b7-b403-a1b9943af0e8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:54:33,889 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:54:33,907 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:54:33,933 DEBUG awx.main.dispatch task c13298ee-ec2b-43b7-b403-a1b9943af0e8 is finished
2021-04-08 10:54:53,401 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(2ecd2255-2e56-447c-a37a-34d8e77524b8, queue=dev-awx-0)
[2021-04-08 10:54:53,401: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(2ecd2255-2e56-447c-a37a-34d8e77524b8, queue=dev-awx-0)
2021-04-08 10:54:53,453 DEBUG awx.main.dispatch delivered 2ecd2255-2e56-447c-a37a-34d8e77524b8 to worker[179] qsize 0
2021-04-08 10:54:53,459 DEBUG awx.main.dispatch task 2ecd2255-2e56-447c-a37a-34d8e77524b8 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 10:54:53,463 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 10:54:53,489 DEBUG awx.main.dispatch task 2ecd2255-2e56-447c-a37a-34d8e77524b8 is finished
2021-04-08 10:54:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(50a61ffc-90e1-4f11-83d4-e19d2a45130e, queue=dev-awx-0)
[2021-04-08 10:54:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(50a61ffc-90e1-4f11-83d4-e19d2a45130e, queue=dev-awx-0)
2021-04-08 10:54:53,640 DEBUG awx.main.dispatch delivered 50a61ffc-90e1-4f11-83d4-e19d2a45130e to worker[178] qsize 0
2021-04-08 10:54:53,643 DEBUG awx.main.dispatch task 50a61ffc-90e1-4f11-83d4-e19d2a45130e starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 10:54:53,661 DEBUG awx.main.dispatch task 50a61ffc-90e1-4f11-83d4-e19d2a45130e is finished
2021-04-08 10:54:53,820 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(548e1c00-a31e-446e-baf2-7bd647538e7c, queue=awx_private_queue)
[2021-04-08 10:54:53,820: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(548e1c00-a31e-446e-baf2-7bd647538e7c, queue=awx_private_queue)
2021-04-08 10:54:53,833 DEBUG awx.main.dispatch delivered 548e1c00-a31e-446e-baf2-7bd647538e7c to worker[176] qsize 0
2021-04-08 10:54:53,836 DEBUG awx.main.dispatch task 548e1c00-a31e-446e-baf2-7bd647538e7c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:54:53,847 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:54:53,851 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:54:23.871442+00:00
2021-04-08 10:54:53,861 DEBUG awx.main.dispatch task 548e1c00-a31e-446e-baf2-7bd647538e7c is finished
2021-04-08 10:54:53,870 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(141fc33f-81bd-4d78-ad7c-ac1eabf5c4fa, queue=awx_private_queue)
[2021-04-08 10:54:53,870: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(141fc33f-81bd-4d78-ad7c-ac1eabf5c4fa, queue=awx_private_queue)
2021-04-08 10:54:53,883 DEBUG awx.main.dispatch delivered 141fc33f-81bd-4d78-ad7c-ac1eabf5c4fa to worker[179] qsize 0
2021-04-08 10:54:53,885 DEBUG awx.main.dispatch task 141fc33f-81bd-4d78-ad7c-ac1eabf5c4fa starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:54:53,887 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:54:53,906 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:54:53,945 DEBUG awx.main.dispatch task 141fc33f-81bd-4d78-ad7c-ac1eabf5c4fa is finished
RESULT 2
OKREADY
2021-04-08 10:55:13,877 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d1d3be45-4c3e-443a-85cb-fd9f8e788359, queue=awx_private_queue)
[2021-04-08 10:55:13,877: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d1d3be45-4c3e-443a-85cb-fd9f8e788359, queue=awx_private_queue)
2021-04-08 10:55:13,890 DEBUG awx.main.dispatch delivered d1d3be45-4c3e-443a-85cb-fd9f8e788359 to worker[179] qsize 0
2021-04-08 10:55:13,892 DEBUG awx.main.dispatch task d1d3be45-4c3e-443a-85cb-fd9f8e788359 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:55:13,893 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:55:13,908 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:55:13,937 DEBUG awx.main.dispatch task d1d3be45-4c3e-443a-85cb-fd9f8e788359 is finished
2021-04-08 10:55:23,828 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0f9c7093-08d6-4910-a6f3-132e530cc50f, queue=awx_private_queue)
[2021-04-08 10:55:23,828: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0f9c7093-08d6-4910-a6f3-132e530cc50f, queue=awx_private_queue)
2021-04-08 10:55:23,850 DEBUG awx.main.dispatch delivered 0f9c7093-08d6-4910-a6f3-132e530cc50f to worker[176] qsize 0
2021-04-08 10:55:23,858 DEBUG awx.main.dispatch task 0f9c7093-08d6-4910-a6f3-132e530cc50f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:55:23,869 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:55:23,874 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:54:53.847681+00:00
2021-04-08 10:55:23,890 DEBUG awx.main.dispatch task 0f9c7093-08d6-4910-a6f3-132e530cc50f is finished
2021-04-08 10:55:33,882 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7eb0467c-b268-41fd-828a-9d64ee4be465, queue=awx_private_queue)
[2021-04-08 10:55:33,882: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7eb0467c-b268-41fd-828a-9d64ee4be465, queue=awx_private_queue)
2021-04-08 10:55:33,933 DEBUG awx.main.dispatch delivered 7eb0467c-b268-41fd-828a-9d64ee4be465 to worker[176] qsize 0
2021-04-08 10:55:33,935 DEBUG awx.main.dispatch task 7eb0467c-b268-41fd-828a-9d64ee4be465 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:55:33,938 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:55:33,958 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:55:33,992 DEBUG awx.main.dispatch task 7eb0467c-b268-41fd-828a-9d64ee4be465 is finished
2021-04-08 10:55:53,402 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(fb66108d-97f6-4ac6-b25c-155f9481fa06, queue=dev-awx-0)
[2021-04-08 10:55:53,402: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(fb66108d-97f6-4ac6-b25c-155f9481fa06, queue=dev-awx-0)
2021-04-08 10:55:53,442 DEBUG awx.main.dispatch delivered fb66108d-97f6-4ac6-b25c-155f9481fa06 to worker[176] qsize 0
2021-04-08 10:55:53,445 DEBUG awx.main.dispatch task fb66108d-97f6-4ac6-b25c-155f9481fa06 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 10:55:53,447 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 10:55:53,476 DEBUG awx.main.dispatch task fb66108d-97f6-4ac6-b25c-155f9481fa06 is finished
2021-04-08 10:55:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(adc5f4e4-5930-465a-afe7-53ddc2b0bd8e, queue=dev-awx-0)
[2021-04-08 10:55:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(adc5f4e4-5930-465a-afe7-53ddc2b0bd8e, queue=dev-awx-0)
2021-04-08 10:55:53,633 DEBUG awx.main.dispatch delivered adc5f4e4-5930-465a-afe7-53ddc2b0bd8e to worker[177] qsize 0
2021-04-08 10:55:53,824 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f4b1ea45-2bef-47cc-b2cd-4e90f85751a5, queue=awx_private_queue)
[2021-04-08 10:55:53,824: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f4b1ea45-2bef-47cc-b2cd-4e90f85751a5, queue=awx_private_queue)
2021-04-08 10:55:53,636 DEBUG awx.main.dispatch task adc5f4e4-5930-465a-afe7-53ddc2b0bd8e starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 10:55:53,845 DEBUG awx.main.dispatch delivered f4b1ea45-2bef-47cc-b2cd-4e90f85751a5 to worker[178] qsize 0
2021-04-08 10:55:53,849 DEBUG awx.main.dispatch task adc5f4e4-5930-465a-afe7-53ddc2b0bd8e is finished
2021-04-08 10:55:53,849 DEBUG awx.main.dispatch task f4b1ea45-2bef-47cc-b2cd-4e90f85751a5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:55:53,860 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:55:53,864 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:55:23.870087+00:00
2021-04-08 10:55:53,878 DEBUG awx.main.dispatch task f4b1ea45-2bef-47cc-b2cd-4e90f85751a5 is finished
2021-04-08 10:55:53,880 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2d1b06a7-5e7b-4c3c-af05-b35691a44f2c, queue=awx_private_queue)
[2021-04-08 10:55:53,880: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2d1b06a7-5e7b-4c3c-af05-b35691a44f2c, queue=awx_private_queue)
2021-04-08 10:55:53,895 DEBUG awx.main.dispatch delivered 2d1b06a7-5e7b-4c3c-af05-b35691a44f2c to worker[178] qsize 0
2021-04-08 10:55:53,897 DEBUG awx.main.dispatch task 2d1b06a7-5e7b-4c3c-af05-b35691a44f2c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:55:53,899 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:55:53,914 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:55:53,940 DEBUG awx.main.dispatch task 2d1b06a7-5e7b-4c3c-af05-b35691a44f2c is finished
RESULT 2
OKREADY
2021-04-08 10:56:13,893 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(978b83ee-886a-4fe8-a043-958480cae08f, queue=awx_private_queue)
[2021-04-08 10:56:13,893: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(978b83ee-886a-4fe8-a043-958480cae08f, queue=awx_private_queue)
2021-04-08 10:56:13,913 DEBUG awx.main.dispatch delivered 978b83ee-886a-4fe8-a043-958480cae08f to worker[178] qsize 0
2021-04-08 10:56:13,917 DEBUG awx.main.dispatch task 978b83ee-886a-4fe8-a043-958480cae08f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:56:13,918 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:56:13,935 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:56:13,975 DEBUG awx.main.dispatch task 978b83ee-886a-4fe8-a043-958480cae08f is finished
2021-04-08 10:56:23,823 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(edabe98d-d85c-4262-8273-10772705e24c, queue=awx_private_queue)
[2021-04-08 10:56:23,823: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(edabe98d-d85c-4262-8273-10772705e24c, queue=awx_private_queue)
2021-04-08 10:56:23,836 DEBUG awx.main.dispatch delivered edabe98d-d85c-4262-8273-10772705e24c to worker[176] qsize 0
2021-04-08 10:56:23,839 DEBUG awx.main.dispatch task edabe98d-d85c-4262-8273-10772705e24c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:56:23,847 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:56:23,850 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:55:53.860553+00:00
2021-04-08 10:56:23,863 DEBUG awx.main.dispatch task edabe98d-d85c-4262-8273-10772705e24c is finished
2021-04-08 10:56:33,893 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8c0a2c7d-0182-43f2-8788-a68ae5988e35, queue=awx_private_queue)
[2021-04-08 10:56:33,893: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8c0a2c7d-0182-43f2-8788-a68ae5988e35, queue=awx_private_queue)
2021-04-08 10:56:33,909 DEBUG awx.main.dispatch delivered 8c0a2c7d-0182-43f2-8788-a68ae5988e35 to worker[177] qsize 0
2021-04-08 10:56:33,911 DEBUG awx.main.dispatch task 8c0a2c7d-0182-43f2-8788-a68ae5988e35 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:56:33,913 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:56:33,933 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:56:33,969 DEBUG awx.main.dispatch task 8c0a2c7d-0182-43f2-8788-a68ae5988e35 is finished
2021-04-08 10:56:53,344 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(f628af50-0e0b-4159-a448-e092f73525d5, queue=dev-awx-0)
[2021-04-08 10:56:53,344: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(f628af50-0e0b-4159-a448-e092f73525d5, queue=dev-awx-0)
2021-04-08 10:56:53,397 DEBUG awx.main.dispatch delivered f628af50-0e0b-4159-a448-e092f73525d5 to worker[176] qsize 0
2021-04-08 10:56:53,403 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(657dae22-353a-4fde-a19a-0ec4b63975bf, queue=dev-awx-0)
[2021-04-08 10:56:53,403: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(657dae22-353a-4fde-a19a-0ec4b63975bf, queue=dev-awx-0)
2021-04-08 10:56:53,434 DEBUG awx.main.dispatch delivered 657dae22-353a-4fde-a19a-0ec4b63975bf to worker[177] qsize 0
2021-04-08 10:56:53,438 DEBUG awx.main.dispatch task 657dae22-353a-4fde-a19a-0ec4b63975bf starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 10:56:53,443 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 10:56:53,474 DEBUG awx.main.dispatch task 657dae22-353a-4fde-a19a-0ec4b63975bf is finished
2021-04-08 10:56:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(0fa335a9-809c-415b-9d52-7b03a1a165d3, queue=dev-awx-0)
[2021-04-08 10:56:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(0fa335a9-809c-415b-9d52-7b03a1a165d3, queue=dev-awx-0)
2021-04-08 10:56:53,404 DEBUG awx.main.dispatch task f628af50-0e0b-4159-a448-e092f73525d5 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-04-08 10:56:53,631 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-04-08 10:56:53,644 DEBUG awx.main.dispatch task f628af50-0e0b-4159-a448-e092f73525d5 is finished
2021-04-08 10:56:53,655 DEBUG awx.main.dispatch delivered 0fa335a9-809c-415b-9d52-7b03a1a165d3 to worker[177] qsize 0
2021-04-08 10:56:53,658 DEBUG awx.main.dispatch task 0fa335a9-809c-415b-9d52-7b03a1a165d3 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 10:56:53,673 DEBUG awx.main.dispatch task 0fa335a9-809c-415b-9d52-7b03a1a165d3 is finished
2021-04-08 10:56:53,823 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(80e1cbb7-145e-453d-b18b-2a5af532872a, queue=awx_private_queue)
[2021-04-08 10:56:53,823: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(80e1cbb7-145e-453d-b18b-2a5af532872a, queue=awx_private_queue)
2021-04-08 10:56:53,850 DEBUG awx.main.dispatch delivered 80e1cbb7-145e-453d-b18b-2a5af532872a to worker[179] qsize 0
2021-04-08 10:56:53,854 DEBUG awx.main.dispatch task 80e1cbb7-145e-453d-b18b-2a5af532872a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:56:53,866 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:56:53,872 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:56:23.847911+00:00
2021-04-08 10:56:53,889 DEBUG awx.main.dispatch task 80e1cbb7-145e-453d-b18b-2a5af532872a is finished
2021-04-08 10:56:53,893 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b60ba703-e208-41ac-a6d2-6b3ed9f20405, queue=awx_private_queue)
[2021-04-08 10:56:53,893: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b60ba703-e208-41ac-a6d2-6b3ed9f20405, queue=awx_private_queue)
2021-04-08 10:56:53,909 DEBUG awx.main.dispatch delivered b60ba703-e208-41ac-a6d2-6b3ed9f20405 to worker[179] qsize 0
2021-04-08 10:56:53,911 DEBUG awx.main.dispatch task b60ba703-e208-41ac-a6d2-6b3ed9f20405 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:56:53,912 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:56:53,925 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:56:53,950 DEBUG awx.main.dispatch task b60ba703-e208-41ac-a6d2-6b3ed9f20405 is finished
RESULT 2
OKREADY
2021-04-08 10:57:13,901 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9c640302-1fba-448b-8ce2-187277498144, queue=awx_private_queue)
[2021-04-08 10:57:13,901: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9c640302-1fba-448b-8ce2-187277498144, queue=awx_private_queue)
2021-04-08 10:57:13,915 DEBUG awx.main.dispatch delivered 9c640302-1fba-448b-8ce2-187277498144 to worker[176] qsize 0
2021-04-08 10:57:13,919 DEBUG awx.main.dispatch task 9c640302-1fba-448b-8ce2-187277498144 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:57:13,920 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:57:13,942 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:57:13,974 DEBUG awx.main.dispatch task 9c640302-1fba-448b-8ce2-187277498144 is finished
2021-04-08 10:57:23,824 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9142639e-3bcb-4d47-9733-6f9b32331060, queue=awx_private_queue)
[2021-04-08 10:57:23,824: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9142639e-3bcb-4d47-9733-6f9b32331060, queue=awx_private_queue)
2021-04-08 10:57:23,840 DEBUG awx.main.dispatch delivered 9142639e-3bcb-4d47-9733-6f9b32331060 to worker[177] qsize 0
2021-04-08 10:57:23,843 DEBUG awx.main.dispatch task 9142639e-3bcb-4d47-9733-6f9b32331060 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:57:23,854 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:57:23,859 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:56:53.867334+00:00
2021-04-08 10:57:23,878 DEBUG awx.main.dispatch task 9142639e-3bcb-4d47-9733-6f9b32331060 is finished
2021-04-08 10:57:33,909 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(570dd1ce-0625-4552-af05-85d6324dc339, queue=awx_private_queue)
[2021-04-08 10:57:33,909: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(570dd1ce-0625-4552-af05-85d6324dc339, queue=awx_private_queue)
2021-04-08 10:57:33,932 DEBUG awx.main.dispatch delivered 570dd1ce-0625-4552-af05-85d6324dc339 to worker[179] qsize 0
2021-04-08 10:57:33,939 DEBUG awx.main.dispatch task 570dd1ce-0625-4552-af05-85d6324dc339 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:57:33,943 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:57:33,978 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:57:34,008 DEBUG awx.main.dispatch task 570dd1ce-0625-4552-af05-85d6324dc339 is finished
2021-04-08 10:57:53,413 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6a0f99b7-1926-4e83-b759-7027ecdfb409, queue=dev-awx-0)
[2021-04-08 10:57:53,413: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6a0f99b7-1926-4e83-b759-7027ecdfb409, queue=dev-awx-0)
2021-04-08 10:57:53,461 DEBUG awx.main.dispatch delivered 6a0f99b7-1926-4e83-b759-7027ecdfb409 to worker[179] qsize 0
2021-04-08 10:57:53,463 DEBUG awx.main.dispatch task 6a0f99b7-1926-4e83-b759-7027ecdfb409 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 10:57:53,464 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 10:57:53,493 DEBUG awx.main.dispatch task 6a0f99b7-1926-4e83-b759-7027ecdfb409 is finished
2021-04-08 10:57:53,619 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(0f051ad7-85b7-47fe-a2d8-dc404ec28aeb, queue=dev-awx-0)
[2021-04-08 10:57:53,619: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(0f051ad7-85b7-47fe-a2d8-dc404ec28aeb, queue=dev-awx-0)
2021-04-08 10:57:53,649 DEBUG awx.main.dispatch delivered 0f051ad7-85b7-47fe-a2d8-dc404ec28aeb to worker[176] qsize 0
2021-04-08 10:57:53,824 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d058d5ff-4cb7-479c-90c5-f1cd1077e73d, queue=awx_private_queue)
[2021-04-08 10:57:53,824: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d058d5ff-4cb7-479c-90c5-f1cd1077e73d, queue=awx_private_queue)
2021-04-08 10:57:53,838 DEBUG awx.main.dispatch delivered d058d5ff-4cb7-479c-90c5-f1cd1077e73d to worker[178] qsize 0
2021-04-08 10:57:53,663 DEBUG awx.main.dispatch task 0f051ad7-85b7-47fe-a2d8-dc404ec28aeb starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 10:57:53,871 DEBUG awx.main.dispatch task 0f051ad7-85b7-47fe-a2d8-dc404ec28aeb is finished
2021-04-08 10:57:53,902 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1a0fbaf4-3df9-45ac-8825-25b200634eac, queue=awx_private_queue)
[2021-04-08 10:57:53,902: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1a0fbaf4-3df9-45ac-8825-25b200634eac, queue=awx_private_queue)
2021-04-08 10:57:53,918 DEBUG awx.main.dispatch delivered 1a0fbaf4-3df9-45ac-8825-25b200634eac to worker[179] qsize 0
2021-04-08 10:57:53,920 DEBUG awx.main.dispatch task 1a0fbaf4-3df9-45ac-8825-25b200634eac starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:57:53,922 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:57:53,933 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:57:53,954 DEBUG awx.main.dispatch task 1a0fbaf4-3df9-45ac-8825-25b200634eac is finished
2021-04-08 10:57:53,841 DEBUG awx.main.dispatch task d058d5ff-4cb7-479c-90c5-f1cd1077e73d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:57:54,042 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:57:54,044 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:57:23.855112+00:00
2021-04-08 10:57:54,057 DEBUG awx.main.dispatch task d058d5ff-4cb7-479c-90c5-f1cd1077e73d is finished
RESULT 2
OKREADY
2021-04-08 10:58:13,914 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1d707640-3376-47b0-a542-43b0f3bee7b8, queue=awx_private_queue)
[2021-04-08 10:58:13,914: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1d707640-3376-47b0-a542-43b0f3bee7b8, queue=awx_private_queue)
2021-04-08 10:58:13,937 DEBUG awx.main.dispatch delivered 1d707640-3376-47b0-a542-43b0f3bee7b8 to worker[179] qsize 0
2021-04-08 10:58:13,945 DEBUG awx.main.dispatch task 1d707640-3376-47b0-a542-43b0f3bee7b8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:58:13,948 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:58:13,970 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:58:13,997 DEBUG awx.main.dispatch task 1d707640-3376-47b0-a542-43b0f3bee7b8 is finished
2021-04-08 10:58:23,827 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6b60d8d6-b389-46fb-a1e7-91ac49cced05, queue=awx_private_queue)
[2021-04-08 10:58:23,827: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6b60d8d6-b389-46fb-a1e7-91ac49cced05, queue=awx_private_queue)
2021-04-08 10:58:23,866 DEBUG awx.main.dispatch delivered 6b60d8d6-b389-46fb-a1e7-91ac49cced05 to worker[177] qsize 0
2021-04-08 10:58:23,876 DEBUG awx.main.dispatch task 6b60d8d6-b389-46fb-a1e7-91ac49cced05 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:58:23,902 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:58:23,909 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:57:54.042699+00:00
2021-04-08 10:58:23,925 DEBUG awx.main.dispatch task 6b60d8d6-b389-46fb-a1e7-91ac49cced05 is finished
2021-04-08 10:58:33,912 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e650977d-66ec-4a49-8678-8f6f5c7d1b1a, queue=awx_private_queue)
[2021-04-08 10:58:33,912: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e650977d-66ec-4a49-8678-8f6f5c7d1b1a, queue=awx_private_queue)
2021-04-08 10:58:33,935 DEBUG awx.main.dispatch delivered e650977d-66ec-4a49-8678-8f6f5c7d1b1a to worker[176] qsize 0
2021-04-08 10:58:33,942 DEBUG awx.main.dispatch task e650977d-66ec-4a49-8678-8f6f5c7d1b1a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:58:33,948 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:58:33,978 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:58:34,005 DEBUG awx.main.dispatch task e650977d-66ec-4a49-8678-8f6f5c7d1b1a is finished
2021-04-08 10:58:53,426 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(22e6ff14-5a7b-4e10-b0f9-fe3156074baa, queue=dev-awx-0)
[2021-04-08 10:58:53,426: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(22e6ff14-5a7b-4e10-b0f9-fe3156074baa, queue=dev-awx-0)
2021-04-08 10:58:53,506 DEBUG awx.main.dispatch delivered 22e6ff14-5a7b-4e10-b0f9-fe3156074baa to worker[178] qsize 0
2021-04-08 10:58:53,516 DEBUG awx.main.dispatch task 22e6ff14-5a7b-4e10-b0f9-fe3156074baa starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 10:58:53,522 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 10:58:53,555 DEBUG awx.main.dispatch task 22e6ff14-5a7b-4e10-b0f9-fe3156074baa is finished
2021-04-08 10:58:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(43b4620a-e3b8-4bfe-a477-fe5d0f2fe617, queue=dev-awx-0)
[2021-04-08 10:58:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(43b4620a-e3b8-4bfe-a477-fe5d0f2fe617, queue=dev-awx-0)
2021-04-08 10:58:53,641 DEBUG awx.main.dispatch delivered 43b4620a-e3b8-4bfe-a477-fe5d0f2fe617 to worker[178] qsize 0
2021-04-08 10:58:53,644 DEBUG awx.main.dispatch task 43b4620a-e3b8-4bfe-a477-fe5d0f2fe617 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 10:58:53,662 DEBUG awx.main.dispatch task 43b4620a-e3b8-4bfe-a477-fe5d0f2fe617 is finished
2021-04-08 10:58:53,824 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(21fadb2c-3a00-448e-a517-d6aace7ab536, queue=awx_private_queue)
[2021-04-08 10:58:53,824: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(21fadb2c-3a00-448e-a517-d6aace7ab536, queue=awx_private_queue)
2021-04-08 10:58:53,838 DEBUG awx.main.dispatch delivered 21fadb2c-3a00-448e-a517-d6aace7ab536 to worker[178] qsize 0
2021-04-08 10:58:53,841 DEBUG awx.main.dispatch task 21fadb2c-3a00-448e-a517-d6aace7ab536 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:58:53,849 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:58:53,852 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:58:23.902856+00:00
2021-04-08 10:58:53,865 DEBUG awx.main.dispatch task 21fadb2c-3a00-448e-a517-d6aace7ab536 is finished
2021-04-08 10:58:53,911 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(97909ff7-5151-4ffe-96ba-cc2fa326be6a, queue=awx_private_queue)
[2021-04-08 10:58:53,911: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(97909ff7-5151-4ffe-96ba-cc2fa326be6a, queue=awx_private_queue)
2021-04-08 10:58:54,121 DEBUG awx.main.dispatch delivered 97909ff7-5151-4ffe-96ba-cc2fa326be6a to worker[177] qsize 0
2021-04-08 10:58:54,125 DEBUG awx.main.dispatch task 97909ff7-5151-4ffe-96ba-cc2fa326be6a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:58:54,128 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:58:54,152 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:58:54,184 DEBUG awx.main.dispatch task 97909ff7-5151-4ffe-96ba-cc2fa326be6a is finished
RESULT 2
OKREADY
2021-04-08 10:59:13,973 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(dfc4dd3b-7d06-4060-9846-3419191c5703, queue=awx_private_queue)
[2021-04-08 10:59:13,973: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(dfc4dd3b-7d06-4060-9846-3419191c5703, queue=awx_private_queue)
2021-04-08 10:59:13,986 DEBUG awx.main.dispatch delivered dfc4dd3b-7d06-4060-9846-3419191c5703 to worker[177] qsize 0
2021-04-08 10:59:13,989 DEBUG awx.main.dispatch task dfc4dd3b-7d06-4060-9846-3419191c5703 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:59:13,990 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:59:14,019 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:59:14,057 DEBUG awx.main.dispatch task dfc4dd3b-7d06-4060-9846-3419191c5703 is finished
2021-04-08 10:59:23,829 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b2b1b731-5e01-4553-bf69-4da7d7553e40, queue=awx_private_queue)
[2021-04-08 10:59:23,829: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b2b1b731-5e01-4553-bf69-4da7d7553e40, queue=awx_private_queue)
2021-04-08 10:59:23,849 DEBUG awx.main.dispatch delivered b2b1b731-5e01-4553-bf69-4da7d7553e40 to worker[177] qsize 0
2021-04-08 10:59:23,855 DEBUG awx.main.dispatch task b2b1b731-5e01-4553-bf69-4da7d7553e40 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:59:23,866 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:59:23,870 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:58:53.849812+00:00
2021-04-08 10:59:23,889 DEBUG awx.main.dispatch task b2b1b731-5e01-4553-bf69-4da7d7553e40 is finished
2021-04-08 10:59:33,975 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c3a4b976-b237-4ba4-b7ba-376856f8e586, queue=awx_private_queue)
[2021-04-08 10:59:33,975: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c3a4b976-b237-4ba4-b7ba-376856f8e586, queue=awx_private_queue)
2021-04-08 10:59:34,002 DEBUG awx.main.dispatch delivered c3a4b976-b237-4ba4-b7ba-376856f8e586 to worker[177] qsize 0
2021-04-08 10:59:34,005 DEBUG awx.main.dispatch task c3a4b976-b237-4ba4-b7ba-376856f8e586 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:59:34,007 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:59:34,030 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:59:34,063 DEBUG awx.main.dispatch task c3a4b976-b237-4ba4-b7ba-376856f8e586 is finished
2021-04-08 10:59:53,431 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(888cd8f9-6800-4108-b329-d6b136821143, queue=dev-awx-0)
[2021-04-08 10:59:53,431: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(888cd8f9-6800-4108-b329-d6b136821143, queue=dev-awx-0)
2021-04-08 10:59:53,521 DEBUG awx.main.dispatch delivered 888cd8f9-6800-4108-b329-d6b136821143 to worker[178] qsize 0
2021-04-08 10:59:53,543 DEBUG awx.main.dispatch task 888cd8f9-6800-4108-b329-d6b136821143 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 10:59:53,548 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 10:59:53,592 DEBUG awx.main.dispatch task 888cd8f9-6800-4108-b329-d6b136821143 is finished
2021-04-08 10:59:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(7473da7a-d46f-4ca6-9f8b-aebfa0753f99, queue=dev-awx-0)
[2021-04-08 10:59:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(7473da7a-d46f-4ca6-9f8b-aebfa0753f99, queue=dev-awx-0)
2021-04-08 10:59:53,649 DEBUG awx.main.dispatch delivered 7473da7a-d46f-4ca6-9f8b-aebfa0753f99 to worker[179] qsize 0
2021-04-08 10:59:53,826 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(37d0f170-b005-49ca-8be4-956694e91a03, queue=awx_private_queue)
[2021-04-08 10:59:53,826: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(37d0f170-b005-49ca-8be4-956694e91a03, queue=awx_private_queue)
2021-04-08 10:59:53,652 DEBUG awx.main.dispatch task 7473da7a-d46f-4ca6-9f8b-aebfa0753f99 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 10:59:53,848 DEBUG awx.main.dispatch task 7473da7a-d46f-4ca6-9f8b-aebfa0753f99 is finished
2021-04-08 10:59:53,863 DEBUG awx.main.dispatch delivered 37d0f170-b005-49ca-8be4-956694e91a03 to worker[178] qsize 0
2021-04-08 10:59:53,868 DEBUG awx.main.dispatch task 37d0f170-b005-49ca-8be4-956694e91a03 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 10:59:53,887 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 10:59:53,890 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:59:23.867246+00:00
2021-04-08 10:59:53,905 DEBUG awx.main.dispatch task 37d0f170-b005-49ca-8be4-956694e91a03 is finished
2021-04-08 10:59:53,972 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8cae7615-d408-435f-bd0f-d5945d906593, queue=awx_private_queue)
[2021-04-08 10:59:53,972: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8cae7615-d408-435f-bd0f-d5945d906593, queue=awx_private_queue)
2021-04-08 10:59:53,985 DEBUG awx.main.dispatch delivered 8cae7615-d408-435f-bd0f-d5945d906593 to worker[176] qsize 0
2021-04-08 10:59:53,989 DEBUG awx.main.dispatch task 8cae7615-d408-435f-bd0f-d5945d906593 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 10:59:53,995 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 10:59:54,022 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 10:59:54,054 DEBUG awx.main.dispatch task 8cae7615-d408-435f-bd0f-d5945d906593 is finished
RESULT 2
OKREADY
2021-04-08 11:00:13,988 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(abb01065-bcfd-48b4-9ee7-dcdd9f2e8e8d, queue=awx_private_queue)
[2021-04-08 11:00:13,988: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(abb01065-bcfd-48b4-9ee7-dcdd9f2e8e8d, queue=awx_private_queue)
2021-04-08 11:00:14,015 DEBUG awx.main.dispatch delivered abb01065-bcfd-48b4-9ee7-dcdd9f2e8e8d to worker[179] qsize 0
2021-04-08 11:00:14,022 DEBUG awx.main.dispatch task abb01065-bcfd-48b4-9ee7-dcdd9f2e8e8d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:00:14,027 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:00:14,061 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:00:14,089 DEBUG awx.main.dispatch task abb01065-bcfd-48b4-9ee7-dcdd9f2e8e8d is finished
2021-04-08 11:00:23,826 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7129e0d2-73c5-4aef-8234-82822daa5a9f, queue=awx_private_queue)
[2021-04-08 11:00:23,826: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7129e0d2-73c5-4aef-8234-82822daa5a9f, queue=awx_private_queue)
2021-04-08 11:00:23,840 DEBUG awx.main.dispatch delivered 7129e0d2-73c5-4aef-8234-82822daa5a9f to worker[176] qsize 0
2021-04-08 11:00:23,844 DEBUG awx.main.dispatch task 7129e0d2-73c5-4aef-8234-82822daa5a9f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:00:23,867 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:00:23,875 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 10:59:53.887968+00:00
2021-04-08 11:00:23,893 DEBUG awx.main.dispatch task 7129e0d2-73c5-4aef-8234-82822daa5a9f is finished
2021-04-08 11:00:33,989 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0024b520-e733-4efb-9ceb-19174802c634, queue=awx_private_queue)
[2021-04-08 11:00:33,989: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0024b520-e733-4efb-9ceb-19174802c634, queue=awx_private_queue)
2021-04-08 11:00:34,033 DEBUG awx.main.dispatch delivered 0024b520-e733-4efb-9ceb-19174802c634 to worker[178] qsize 0
2021-04-08 11:00:34,046 DEBUG awx.main.dispatch task 0024b520-e733-4efb-9ceb-19174802c634 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:00:34,061 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:00:34,103 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:00:34,129 DEBUG awx.main.dispatch task 0024b520-e733-4efb-9ceb-19174802c634 is finished
2021-04-08 11:00:53,437 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6f590bf2-1b95-4421-9cad-58ac8098998f, queue=dev-awx-0)
[2021-04-08 11:00:53,437: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6f590bf2-1b95-4421-9cad-58ac8098998f, queue=dev-awx-0)
2021-04-08 11:00:53,486 DEBUG awx.main.dispatch delivered 6f590bf2-1b95-4421-9cad-58ac8098998f to worker[179] qsize 0
2021-04-08 11:00:53,489 DEBUG awx.main.dispatch task 6f590bf2-1b95-4421-9cad-58ac8098998f starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:00:53,493 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:00:53,518 DEBUG awx.main.dispatch task 6f590bf2-1b95-4421-9cad-58ac8098998f is finished
2021-04-08 11:00:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(297555f6-4ba8-4df8-91ec-eb9b114998c5, queue=dev-awx-0)
[2021-04-08 11:00:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(297555f6-4ba8-4df8-91ec-eb9b114998c5, queue=dev-awx-0)
2021-04-08 11:00:53,637 DEBUG awx.main.dispatch delivered 297555f6-4ba8-4df8-91ec-eb9b114998c5 to worker[179] qsize 0
2021-04-08 11:00:53,639 DEBUG awx.main.dispatch task 297555f6-4ba8-4df8-91ec-eb9b114998c5 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:00:53,826 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(be7aeaa4-87bc-4962-a32b-ac29167cafa6, queue=awx_private_queue)
[2021-04-08 11:00:53,826: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(be7aeaa4-87bc-4962-a32b-ac29167cafa6, queue=awx_private_queue)
2021-04-08 11:00:53,844 DEBUG awx.main.dispatch delivered be7aeaa4-87bc-4962-a32b-ac29167cafa6 to worker[176] qsize 0
2021-04-08 11:00:53,848 DEBUG awx.main.dispatch task be7aeaa4-87bc-4962-a32b-ac29167cafa6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:00:53,655 DEBUG awx.main.dispatch task 297555f6-4ba8-4df8-91ec-eb9b114998c5 is finished
2021-04-08 11:00:53,859 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:00:53,863 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:00:23.867933+00:00
2021-04-08 11:00:53,875 DEBUG awx.main.dispatch task be7aeaa4-87bc-4962-a32b-ac29167cafa6 is finished
2021-04-08 11:00:53,983 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ce173cb6-1adb-430f-9949-a1048f6718fb, queue=awx_private_queue)
[2021-04-08 11:00:53,983: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ce173cb6-1adb-430f-9949-a1048f6718fb, queue=awx_private_queue)
2021-04-08 11:00:54,006 DEBUG awx.main.dispatch delivered ce173cb6-1adb-430f-9949-a1048f6718fb to worker[176] qsize 0
2021-04-08 11:00:54,008 DEBUG awx.main.dispatch task ce173cb6-1adb-430f-9949-a1048f6718fb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:00:54,010 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:00:54,028 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:00:54,054 DEBUG awx.main.dispatch task ce173cb6-1adb-430f-9949-a1048f6718fb is finished
RESULT 2
OKREADY
2021-04-08 11:01:13,994 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e6539e51-721c-495c-82c3-887a39bce20d, queue=awx_private_queue)
[2021-04-08 11:01:13,994: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e6539e51-721c-495c-82c3-887a39bce20d, queue=awx_private_queue)
2021-04-08 11:01:14,021 DEBUG awx.main.dispatch delivered e6539e51-721c-495c-82c3-887a39bce20d to worker[177] qsize 0
2021-04-08 11:01:14,031 DEBUG awx.main.dispatch task e6539e51-721c-495c-82c3-887a39bce20d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:01:14,036 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:01:14,060 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:01:14,088 DEBUG awx.main.dispatch task e6539e51-721c-495c-82c3-887a39bce20d is finished
2021-04-08 11:01:23,828 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9bfd8651-6fac-4755-80b5-32b4e1cccf16, queue=awx_private_queue)
[2021-04-08 11:01:23,828: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9bfd8651-6fac-4755-80b5-32b4e1cccf16, queue=awx_private_queue)
2021-04-08 11:01:23,845 DEBUG awx.main.dispatch delivered 9bfd8651-6fac-4755-80b5-32b4e1cccf16 to worker[177] qsize 0
2021-04-08 11:01:23,849 DEBUG awx.main.dispatch task 9bfd8651-6fac-4755-80b5-32b4e1cccf16 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:01:23,860 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:01:23,864 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:00:53.859783+00:00
2021-04-08 11:01:23,884 DEBUG awx.main.dispatch task 9bfd8651-6fac-4755-80b5-32b4e1cccf16 is finished
2021-04-08 11:01:33,990 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a264e1b1-8b16-4453-ad73-29fb0faba0e4, queue=awx_private_queue)
[2021-04-08 11:01:33,990: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a264e1b1-8b16-4453-ad73-29fb0faba0e4, queue=awx_private_queue)
2021-04-08 11:01:34,005 DEBUG awx.main.dispatch delivered a264e1b1-8b16-4453-ad73-29fb0faba0e4 to worker[176] qsize 0
2021-04-08 11:01:34,010 DEBUG awx.main.dispatch task a264e1b1-8b16-4453-ad73-29fb0faba0e4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:01:34,014 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:01:34,044 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:01:34,074 DEBUG awx.main.dispatch task a264e1b1-8b16-4453-ad73-29fb0faba0e4 is finished
2021-04-08 11:01:53,437 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(0cdebee0-afc1-42d7-9f63-1c94775fd922, queue=dev-awx-0)
[2021-04-08 11:01:53,437: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(0cdebee0-afc1-42d7-9f63-1c94775fd922, queue=dev-awx-0)
2021-04-08 11:01:53,493 DEBUG awx.main.dispatch delivered 0cdebee0-afc1-42d7-9f63-1c94775fd922 to worker[176] qsize 0
2021-04-08 11:01:53,496 DEBUG awx.main.dispatch task 0cdebee0-afc1-42d7-9f63-1c94775fd922 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:01:53,498 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:01:53,520 DEBUG awx.main.dispatch task 0cdebee0-afc1-42d7-9f63-1c94775fd922 is finished
2021-04-08 11:01:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a4d5b8ff-1197-437b-8a33-8cf2a2e9cd6a, queue=dev-awx-0)
[2021-04-08 11:01:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a4d5b8ff-1197-437b-8a33-8cf2a2e9cd6a, queue=dev-awx-0)
2021-04-08 11:01:53,634 DEBUG awx.main.dispatch delivered a4d5b8ff-1197-437b-8a33-8cf2a2e9cd6a to worker[176] qsize 0
2021-04-08 11:01:53,636 DEBUG awx.main.dispatch task a4d5b8ff-1197-437b-8a33-8cf2a2e9cd6a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:01:53,647 DEBUG awx.main.dispatch task a4d5b8ff-1197-437b-8a33-8cf2a2e9cd6a is finished
2021-04-08 11:01:53,828 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(70203223-48c2-410a-af45-4a63e34547c8, queue=awx_private_queue)
[2021-04-08 11:01:53,828: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(70203223-48c2-410a-af45-4a63e34547c8, queue=awx_private_queue)
2021-04-08 11:01:53,846 DEBUG awx.main.dispatch delivered 70203223-48c2-410a-af45-4a63e34547c8 to worker[178] qsize 0
2021-04-08 11:01:53,851 DEBUG awx.main.dispatch task 70203223-48c2-410a-af45-4a63e34547c8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:01:54,056 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:01:54,059 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:01:23.860772+00:00
2021-04-08 11:01:54,070 DEBUG awx.main.dispatch task 70203223-48c2-410a-af45-4a63e34547c8 is finished
2021-04-08 11:01:53,991 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4a37e1ea-be19-4247-a33f-716cdf19d870, queue=awx_private_queue)
[2021-04-08 11:01:53,991: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4a37e1ea-be19-4247-a33f-716cdf19d870, queue=awx_private_queue)
2021-04-08 11:01:54,185 DEBUG awx.main.dispatch delivered 4a37e1ea-be19-4247-a33f-716cdf19d870 to worker[176] qsize 0
2021-04-08 11:01:54,187 DEBUG awx.main.dispatch task 4a37e1ea-be19-4247-a33f-716cdf19d870 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:01:54,189 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:01:54,210 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:01:54,235 DEBUG awx.main.dispatch task 4a37e1ea-be19-4247-a33f-716cdf19d870 is finished
RESULT 2
OKREADY
2021-04-08 11:02:14,008 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(706cdbf0-6742-451b-9d2e-5e76092fe586, queue=awx_private_queue)
[2021-04-08 11:02:14,008: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(706cdbf0-6742-451b-9d2e-5e76092fe586, queue=awx_private_queue)
2021-04-08 11:02:14,030 DEBUG awx.main.dispatch delivered 706cdbf0-6742-451b-9d2e-5e76092fe586 to worker[178] qsize 0
2021-04-08 11:02:14,034 DEBUG awx.main.dispatch task 706cdbf0-6742-451b-9d2e-5e76092fe586 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:02:14,036 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:02:14,064 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:02:14,099 DEBUG awx.main.dispatch task 706cdbf0-6742-451b-9d2e-5e76092fe586 is finished
2021-04-08 11:02:23,831 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(baf0b97f-ae58-4937-9208-8a77e8e35d4e, queue=awx_private_queue)
[2021-04-08 11:02:23,831: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(baf0b97f-ae58-4937-9208-8a77e8e35d4e, queue=awx_private_queue)
2021-04-08 11:02:23,860 DEBUG awx.main.dispatch delivered baf0b97f-ae58-4937-9208-8a77e8e35d4e to worker[176] qsize 0
2021-04-08 11:02:23,869 DEBUG awx.main.dispatch task baf0b97f-ae58-4937-9208-8a77e8e35d4e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:02:23,884 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:02:23,888 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:01:54.057024+00:00
2021-04-08 11:02:23,904 DEBUG awx.main.dispatch task baf0b97f-ae58-4937-9208-8a77e8e35d4e is finished
2021-04-08 11:02:34,010 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9555413a-b9dc-4abd-ae3d-28bf31891fd1, queue=awx_private_queue)
[2021-04-08 11:02:34,010: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9555413a-b9dc-4abd-ae3d-28bf31891fd1, queue=awx_private_queue)
2021-04-08 11:02:34,035 DEBUG awx.main.dispatch delivered 9555413a-b9dc-4abd-ae3d-28bf31891fd1 to worker[176] qsize 0
2021-04-08 11:02:34,038 DEBUG awx.main.dispatch task 9555413a-b9dc-4abd-ae3d-28bf31891fd1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:02:34,041 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:02:34,063 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:02:34,088 DEBUG awx.main.dispatch task 9555413a-b9dc-4abd-ae3d-28bf31891fd1 is finished
2021-04-08 11:02:53,440 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c71986e4-d5dd-4586-bb91-6c2666fac0c2, queue=dev-awx-0)
[2021-04-08 11:02:53,440: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c71986e4-d5dd-4586-bb91-6c2666fac0c2, queue=dev-awx-0)
2021-04-08 11:02:53,478 DEBUG awx.main.dispatch delivered c71986e4-d5dd-4586-bb91-6c2666fac0c2 to worker[178] qsize 0
2021-04-08 11:02:53,484 DEBUG awx.main.dispatch task c71986e4-d5dd-4586-bb91-6c2666fac0c2 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:02:53,490 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:02:53,520 DEBUG awx.main.dispatch task c71986e4-d5dd-4586-bb91-6c2666fac0c2 is finished
2021-04-08 11:02:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(953555a1-8c3c-479a-87fe-9b764e24be30, queue=dev-awx-0)
[2021-04-08 11:02:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(953555a1-8c3c-479a-87fe-9b764e24be30, queue=dev-awx-0)
2021-04-08 11:02:53,632 DEBUG awx.main.dispatch delivered 953555a1-8c3c-479a-87fe-9b764e24be30 to worker[179] qsize 0
2021-04-08 11:02:53,827 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0587d369-7eb3-4af0-9f39-92a99c82aeec, queue=awx_private_queue)
[2021-04-08 11:02:53,827: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0587d369-7eb3-4af0-9f39-92a99c82aeec, queue=awx_private_queue)
2021-04-08 11:02:53,849 DEBUG awx.main.dispatch delivered 0587d369-7eb3-4af0-9f39-92a99c82aeec to worker[176] qsize 0
2021-04-08 11:02:53,851 DEBUG awx.main.dispatch task 0587d369-7eb3-4af0-9f39-92a99c82aeec starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:02:53,862 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:02:53,865 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:02:23.884261+00:00
2021-04-08 11:02:53,881 DEBUG awx.main.dispatch task 0587d369-7eb3-4af0-9f39-92a99c82aeec is finished
2021-04-08 11:02:53,639 DEBUG awx.main.dispatch task 953555a1-8c3c-479a-87fe-9b764e24be30 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:02:53,897 DEBUG awx.main.dispatch task 953555a1-8c3c-479a-87fe-9b764e24be30 is finished
2021-04-08 11:02:54,007 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(32f14619-c79f-4a05-8006-b1f574652e06, queue=awx_private_queue)
[2021-04-08 11:02:54,007: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(32f14619-c79f-4a05-8006-b1f574652e06, queue=awx_private_queue)
2021-04-08 11:02:54,022 DEBUG awx.main.dispatch delivered 32f14619-c79f-4a05-8006-b1f574652e06 to worker[177] qsize 0
2021-04-08 11:02:54,026 DEBUG awx.main.dispatch task 32f14619-c79f-4a05-8006-b1f574652e06 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:02:54,030 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:02:54,053 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:02:54,081 DEBUG awx.main.dispatch task 32f14619-c79f-4a05-8006-b1f574652e06 is finished
RESULT 2
OKREADY
2021-04-08 11:03:14,007 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8412d7f2-dddf-4330-ad4a-1af0f8f602d8, queue=awx_private_queue)
[2021-04-08 11:03:14,007: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8412d7f2-dddf-4330-ad4a-1af0f8f602d8, queue=awx_private_queue)
2021-04-08 11:03:14,023 DEBUG awx.main.dispatch delivered 8412d7f2-dddf-4330-ad4a-1af0f8f602d8 to worker[177] qsize 0
2021-04-08 11:03:14,025 DEBUG awx.main.dispatch task 8412d7f2-dddf-4330-ad4a-1af0f8f602d8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:03:14,027 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:03:14,041 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:03:14,068 DEBUG awx.main.dispatch task 8412d7f2-dddf-4330-ad4a-1af0f8f602d8 is finished
2021-04-08 11:03:23,828 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f09562f0-ea16-4745-b8f0-52b6f8d19375, queue=awx_private_queue)
[2021-04-08 11:03:23,828: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f09562f0-ea16-4745-b8f0-52b6f8d19375, queue=awx_private_queue)
2021-04-08 11:03:23,849 DEBUG awx.main.dispatch delivered f09562f0-ea16-4745-b8f0-52b6f8d19375 to worker[178] qsize 0
2021-04-08 11:03:23,851 DEBUG awx.main.dispatch task f09562f0-ea16-4745-b8f0-52b6f8d19375 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:03:23,861 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:03:23,863 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:02:53.862578+00:00
2021-04-08 11:03:23,878 DEBUG awx.main.dispatch task f09562f0-ea16-4745-b8f0-52b6f8d19375 is finished
2021-04-08 11:03:34,014 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d4c79b81-ac59-4eaf-8287-53677dc3ad80, queue=awx_private_queue)
[2021-04-08 11:03:34,014: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d4c79b81-ac59-4eaf-8287-53677dc3ad80, queue=awx_private_queue)
2021-04-08 11:03:34,048 DEBUG awx.main.dispatch delivered d4c79b81-ac59-4eaf-8287-53677dc3ad80 to worker[179] qsize 0
2021-04-08 11:03:34,055 DEBUG awx.main.dispatch task d4c79b81-ac59-4eaf-8287-53677dc3ad80 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:03:34,059 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:03:34,084 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:03:34,117 DEBUG awx.main.dispatch task d4c79b81-ac59-4eaf-8287-53677dc3ad80 is finished
2021-04-08 11:03:53,440 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(24d796ed-858c-43aa-a772-feae90685162, queue=dev-awx-0)
[2021-04-08 11:03:53,440: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(24d796ed-858c-43aa-a772-feae90685162, queue=dev-awx-0)
2021-04-08 11:03:53,472 DEBUG awx.main.dispatch delivered 24d796ed-858c-43aa-a772-feae90685162 to worker[179] qsize 0
2021-04-08 11:03:53,474 DEBUG awx.main.dispatch task 24d796ed-858c-43aa-a772-feae90685162 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:03:53,475 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:03:53,499 DEBUG awx.main.dispatch task 24d796ed-858c-43aa-a772-feae90685162 is finished
2021-04-08 11:03:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(773dc773-f1e1-47e1-9358-c038f9e9c299, queue=dev-awx-0)
[2021-04-08 11:03:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(773dc773-f1e1-47e1-9358-c038f9e9c299, queue=dev-awx-0)
2021-04-08 11:03:53,647 DEBUG awx.main.dispatch delivered 773dc773-f1e1-47e1-9358-c038f9e9c299 to worker[177] qsize 0
2021-04-08 11:03:53,651 DEBUG awx.main.dispatch task 773dc773-f1e1-47e1-9358-c038f9e9c299 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:03:53,671 DEBUG awx.main.dispatch task 773dc773-f1e1-47e1-9358-c038f9e9c299 is finished
2021-04-08 11:03:53,827 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(283abc91-f55e-422d-8a53-cbaba06b1549, queue=awx_private_queue)
[2021-04-08 11:03:53,827: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(283abc91-f55e-422d-8a53-cbaba06b1549, queue=awx_private_queue)
2021-04-08 11:03:53,840 DEBUG awx.main.dispatch delivered 283abc91-f55e-422d-8a53-cbaba06b1549 to worker[178] qsize 0
2021-04-08 11:03:53,843 DEBUG awx.main.dispatch task 283abc91-f55e-422d-8a53-cbaba06b1549 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:03:53,854 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:03:53,858 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:03:23.861174+00:00
2021-04-08 11:03:53,871 DEBUG awx.main.dispatch task 283abc91-f55e-422d-8a53-cbaba06b1549 is finished
2021-04-08 11:03:54,009 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(74a3f5df-cd13-4cd9-9ee3-677341214c9e, queue=awx_private_queue)
[2021-04-08 11:03:54,009: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(74a3f5df-cd13-4cd9-9ee3-677341214c9e, queue=awx_private_queue)
2021-04-08 11:03:54,203 DEBUG awx.main.dispatch delivered 74a3f5df-cd13-4cd9-9ee3-677341214c9e to worker[178] qsize 0
2021-04-08 11:03:54,207 DEBUG awx.main.dispatch task 74a3f5df-cd13-4cd9-9ee3-677341214c9e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:03:54,208 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:03:54,224 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:03:54,247 DEBUG awx.main.dispatch task 74a3f5df-cd13-4cd9-9ee3-677341214c9e is finished
RESULT 2
OKREADY
2021-04-08 11:04:14,016 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(94ed934a-216e-4976-be68-118f88a4b3db, queue=awx_private_queue)
[2021-04-08 11:04:14,016: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(94ed934a-216e-4976-be68-118f88a4b3db, queue=awx_private_queue)
2021-04-08 11:04:14,034 DEBUG awx.main.dispatch delivered 94ed934a-216e-4976-be68-118f88a4b3db to worker[176] qsize 0
2021-04-08 11:04:14,038 DEBUG awx.main.dispatch task 94ed934a-216e-4976-be68-118f88a4b3db starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:04:14,042 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:04:14,059 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:04:14,091 DEBUG awx.main.dispatch task 94ed934a-216e-4976-be68-118f88a4b3db is finished
2021-04-08 11:04:23,827 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(156df034-c62e-48ab-be90-54c1e79f44fe, queue=awx_private_queue)
[2021-04-08 11:04:23,827: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(156df034-c62e-48ab-be90-54c1e79f44fe, queue=awx_private_queue)
2021-04-08 11:04:23,840 DEBUG awx.main.dispatch delivered 156df034-c62e-48ab-be90-54c1e79f44fe to worker[179] qsize 0
2021-04-08 11:04:23,842 DEBUG awx.main.dispatch task 156df034-c62e-48ab-be90-54c1e79f44fe starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:04:23,860 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:04:23,862 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:03:53.855137+00:00
2021-04-08 11:04:23,878 DEBUG awx.main.dispatch task 156df034-c62e-48ab-be90-54c1e79f44fe is finished
2021-04-08 11:04:34,016 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e0fa7b06-1e46-474f-8e55-2b222d343ee9, queue=awx_private_queue)
[2021-04-08 11:04:34,016: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e0fa7b06-1e46-474f-8e55-2b222d343ee9, queue=awx_private_queue)
2021-04-08 11:04:34,045 DEBUG awx.main.dispatch delivered e0fa7b06-1e46-474f-8e55-2b222d343ee9 to worker[178] qsize 0
2021-04-08 11:04:34,048 DEBUG awx.main.dispatch task e0fa7b06-1e46-474f-8e55-2b222d343ee9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:04:34,050 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:04:34,069 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:04:34,097 DEBUG awx.main.dispatch task e0fa7b06-1e46-474f-8e55-2b222d343ee9 is finished
2021-04-08 11:04:53,448 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b3bd2240-cdda-40c9-a9e8-81df4bb685ab, queue=dev-awx-0)
[2021-04-08 11:04:53,448: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b3bd2240-cdda-40c9-a9e8-81df4bb685ab, queue=dev-awx-0)
2021-04-08 11:04:53,520 DEBUG awx.main.dispatch delivered b3bd2240-cdda-40c9-a9e8-81df4bb685ab to worker[177] qsize 0
2021-04-08 11:04:53,619 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(4797bdf4-8314-4ea6-b999-8b45dd97f154, queue=dev-awx-0)
[2021-04-08 11:04:53,619: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(4797bdf4-8314-4ea6-b999-8b45dd97f154, queue=dev-awx-0)
2021-04-08 11:04:53,641 DEBUG awx.main.dispatch delivered 4797bdf4-8314-4ea6-b999-8b45dd97f154 to worker[178] qsize 0
2021-04-08 11:04:53,645 DEBUG awx.main.dispatch task 4797bdf4-8314-4ea6-b999-8b45dd97f154 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:04:53,667 DEBUG awx.main.dispatch task 4797bdf4-8314-4ea6-b999-8b45dd97f154 is finished
2021-04-08 11:04:53,525 DEBUG awx.main.dispatch task b3bd2240-cdda-40c9-a9e8-81df4bb685ab starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:04:53,741 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:04:53,764 DEBUG awx.main.dispatch task b3bd2240-cdda-40c9-a9e8-81df4bb685ab is finished
2021-04-08 11:04:53,827 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(09112c0d-9c8b-437b-9405-4421d9074341, queue=awx_private_queue)
[2021-04-08 11:04:53,827: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(09112c0d-9c8b-437b-9405-4421d9074341, queue=awx_private_queue)
2021-04-08 11:04:53,841 DEBUG awx.main.dispatch delivered 09112c0d-9c8b-437b-9405-4421d9074341 to worker[177] qsize 0
2021-04-08 11:04:53,844 DEBUG awx.main.dispatch task 09112c0d-9c8b-437b-9405-4421d9074341 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:04:53,852 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:04:53,854 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:04:23.860460+00:00
2021-04-08 11:04:53,867 DEBUG awx.main.dispatch task 09112c0d-9c8b-437b-9405-4421d9074341 is finished
2021-04-08 11:04:54,015 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7264f77d-571e-4cfc-9b88-a4714e386fa3, queue=awx_private_queue)
[2021-04-08 11:04:54,015: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7264f77d-571e-4cfc-9b88-a4714e386fa3, queue=awx_private_queue)
2021-04-08 11:04:54,028 DEBUG awx.main.dispatch delivered 7264f77d-571e-4cfc-9b88-a4714e386fa3 to worker[178] qsize 0
2021-04-08 11:04:54,030 DEBUG awx.main.dispatch task 7264f77d-571e-4cfc-9b88-a4714e386fa3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:04:54,031 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:04:54,045 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:04:54,070 DEBUG awx.main.dispatch task 7264f77d-571e-4cfc-9b88-a4714e386fa3 is finished
RESULT 2
OKREADY
2021-04-08 11:05:14,029 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f3a2694f-edbc-4e0f-834c-0b08cea46657, queue=awx_private_queue)
[2021-04-08 11:05:14,029: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f3a2694f-edbc-4e0f-834c-0b08cea46657, queue=awx_private_queue)
2021-04-08 11:05:14,058 DEBUG awx.main.dispatch delivered f3a2694f-edbc-4e0f-834c-0b08cea46657 to worker[176] qsize 0
2021-04-08 11:05:14,063 DEBUG awx.main.dispatch task f3a2694f-edbc-4e0f-834c-0b08cea46657 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:05:14,069 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:05:14,089 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:05:14,115 DEBUG awx.main.dispatch task f3a2694f-edbc-4e0f-834c-0b08cea46657 is finished
2021-04-08 11:05:23,827 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3d191194-223c-4d0e-8e2e-7cc414a9637a, queue=awx_private_queue)
[2021-04-08 11:05:23,827: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3d191194-223c-4d0e-8e2e-7cc414a9637a, queue=awx_private_queue)
2021-04-08 11:05:23,842 DEBUG awx.main.dispatch delivered 3d191194-223c-4d0e-8e2e-7cc414a9637a to worker[176] qsize 0
2021-04-08 11:05:23,844 DEBUG awx.main.dispatch task 3d191194-223c-4d0e-8e2e-7cc414a9637a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:05:23,855 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:05:23,859 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:04:53.852876+00:00
2021-04-08 11:05:23,873 DEBUG awx.main.dispatch task 3d191194-223c-4d0e-8e2e-7cc414a9637a is finished
2021-04-08 11:05:34,029 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2b4da871-721b-442b-a376-1528358caa65, queue=awx_private_queue)
[2021-04-08 11:05:34,029: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2b4da871-721b-442b-a376-1528358caa65, queue=awx_private_queue)
2021-04-08 11:05:34,049 DEBUG awx.main.dispatch delivered 2b4da871-721b-442b-a376-1528358caa65 to worker[178] qsize 0
2021-04-08 11:05:34,053 DEBUG awx.main.dispatch task 2b4da871-721b-442b-a376-1528358caa65 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:05:34,056 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:05:34,076 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:05:34,104 DEBUG awx.main.dispatch task 2b4da871-721b-442b-a376-1528358caa65 is finished
2021-04-08 11:05:53,440 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a1cd3ef1-dff5-490c-b9d6-87e1075c08ab, queue=dev-awx-0)
[2021-04-08 11:05:53,440: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a1cd3ef1-dff5-490c-b9d6-87e1075c08ab, queue=dev-awx-0)
2021-04-08 11:05:53,469 DEBUG awx.main.dispatch delivered a1cd3ef1-dff5-490c-b9d6-87e1075c08ab to worker[176] qsize 0
2021-04-08 11:05:53,471 DEBUG awx.main.dispatch task a1cd3ef1-dff5-490c-b9d6-87e1075c08ab starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:05:53,473 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:05:53,494 DEBUG awx.main.dispatch task a1cd3ef1-dff5-490c-b9d6-87e1075c08ab is finished
2021-04-08 11:05:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a1553199-39cb-48f4-9ad5-880dfe8e859d, queue=dev-awx-0)
[2021-04-08 11:05:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a1553199-39cb-48f4-9ad5-880dfe8e859d, queue=dev-awx-0)
2021-04-08 11:05:53,634 DEBUG awx.main.dispatch delivered a1553199-39cb-48f4-9ad5-880dfe8e859d to worker[177] qsize 0
2021-04-08 11:05:53,636 DEBUG awx.main.dispatch task a1553199-39cb-48f4-9ad5-880dfe8e859d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:05:53,649 DEBUG awx.main.dispatch task a1553199-39cb-48f4-9ad5-880dfe8e859d is finished
2021-04-08 11:05:53,828 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(82295d01-a820-4454-b20f-f2c32831d2a4, queue=awx_private_queue)
[2021-04-08 11:05:53,828: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(82295d01-a820-4454-b20f-f2c32831d2a4, queue=awx_private_queue)
2021-04-08 11:05:53,853 DEBUG awx.main.dispatch delivered 82295d01-a820-4454-b20f-f2c32831d2a4 to worker[177] qsize 0
2021-04-08 11:05:53,855 DEBUG awx.main.dispatch task 82295d01-a820-4454-b20f-f2c32831d2a4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:05:53,864 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:05:53,867 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:05:23.855480+00:00
2021-04-08 11:05:53,880 DEBUG awx.main.dispatch task 82295d01-a820-4454-b20f-f2c32831d2a4 is finished
2021-04-08 11:05:54,030 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(486eede8-8737-4641-9cb9-ebdda0238297, queue=awx_private_queue)
[2021-04-08 11:05:54,030: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(486eede8-8737-4641-9cb9-ebdda0238297, queue=awx_private_queue)
2021-04-08 11:05:54,212 DEBUG awx.main.dispatch delivered 486eede8-8737-4641-9cb9-ebdda0238297 to worker[177] qsize 0
2021-04-08 11:05:54,214 DEBUG awx.main.dispatch task 486eede8-8737-4641-9cb9-ebdda0238297 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:05:54,216 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:05:54,229 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:05:54,251 DEBUG awx.main.dispatch task 486eede8-8737-4641-9cb9-ebdda0238297 is finished
RESULT 2
OKREADY
2021-04-08 11:06:14,052 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(66a4350d-3e0a-49a5-aecc-1c970434829f, queue=awx_private_queue)
[2021-04-08 11:06:14,052: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(66a4350d-3e0a-49a5-aecc-1c970434829f, queue=awx_private_queue)
2021-04-08 11:06:14,088 DEBUG awx.main.dispatch delivered 66a4350d-3e0a-49a5-aecc-1c970434829f to worker[176] qsize 0
2021-04-08 11:06:14,094 DEBUG awx.main.dispatch task 66a4350d-3e0a-49a5-aecc-1c970434829f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:06:14,097 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:06:14,117 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:06:14,140 DEBUG awx.main.dispatch task 66a4350d-3e0a-49a5-aecc-1c970434829f is finished
2021-04-08 11:06:23,829 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d2ba2247-0b90-4cf1-bb15-ff8260357337, queue=awx_private_queue)
[2021-04-08 11:06:23,829: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d2ba2247-0b90-4cf1-bb15-ff8260357337, queue=awx_private_queue)
2021-04-08 11:06:23,850 DEBUG awx.main.dispatch delivered d2ba2247-0b90-4cf1-bb15-ff8260357337 to worker[177] qsize 0
2021-04-08 11:06:23,855 DEBUG awx.main.dispatch task d2ba2247-0b90-4cf1-bb15-ff8260357337 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:06:23,870 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:06:23,875 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:05:53.864993+00:00
2021-04-08 11:06:23,892 DEBUG awx.main.dispatch task d2ba2247-0b90-4cf1-bb15-ff8260357337 is finished
2021-04-08 11:06:34,046 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d9874f97-6915-43b8-816d-d5a3ad513fd3, queue=awx_private_queue)
[2021-04-08 11:06:34,046: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d9874f97-6915-43b8-816d-d5a3ad513fd3, queue=awx_private_queue)
2021-04-08 11:06:34,067 DEBUG awx.main.dispatch delivered d9874f97-6915-43b8-816d-d5a3ad513fd3 to worker[177] qsize 0
2021-04-08 11:06:34,071 DEBUG awx.main.dispatch task d9874f97-6915-43b8-816d-d5a3ad513fd3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:06:34,073 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:06:34,106 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:06:34,137 DEBUG awx.main.dispatch task d9874f97-6915-43b8-816d-d5a3ad513fd3 is finished
2021-04-08 11:06:53,337 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(d20c00ea-c70e-45eb-9f22-7f9ed290fdbc, queue=dev-awx-0)
[2021-04-08 11:06:53,337: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(d20c00ea-c70e-45eb-9f22-7f9ed290fdbc, queue=dev-awx-0)
2021-04-08 11:06:53,350 DEBUG awx.main.dispatch delivered d20c00ea-c70e-45eb-9f22-7f9ed290fdbc to worker[177] qsize 0
2021-04-08 11:06:53,352 DEBUG awx.main.dispatch task d20c00ea-c70e-45eb-9f22-7f9ed290fdbc starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-04-08 11:06:53,354 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-04-08 11:06:53,369 DEBUG awx.main.dispatch task d20c00ea-c70e-45eb-9f22-7f9ed290fdbc is finished
2021-04-08 11:06:53,440 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e30c2918-33d6-479d-a4a2-f39aa4d65a38, queue=dev-awx-0)
[2021-04-08 11:06:53,440: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e30c2918-33d6-479d-a4a2-f39aa4d65a38, queue=dev-awx-0)
2021-04-08 11:06:53,471 DEBUG awx.main.dispatch delivered e30c2918-33d6-479d-a4a2-f39aa4d65a38 to worker[177] qsize 0
2021-04-08 11:06:53,473 DEBUG awx.main.dispatch task e30c2918-33d6-479d-a4a2-f39aa4d65a38 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:06:53,474 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:06:53,493 DEBUG awx.main.dispatch task e30c2918-33d6-479d-a4a2-f39aa4d65a38 is finished
2021-04-08 11:06:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(6086d864-4728-4930-9ea3-e25d1dffff7d, queue=dev-awx-0)
[2021-04-08 11:06:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(6086d864-4728-4930-9ea3-e25d1dffff7d, queue=dev-awx-0)
2021-04-08 11:06:53,636 DEBUG awx.main.dispatch delivered 6086d864-4728-4930-9ea3-e25d1dffff7d to worker[178] qsize 0
2021-04-08 11:06:53,640 DEBUG awx.main.dispatch task 6086d864-4728-4930-9ea3-e25d1dffff7d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:06:53,656 DEBUG awx.main.dispatch task 6086d864-4728-4930-9ea3-e25d1dffff7d is finished
2021-04-08 11:06:53,828 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ecdb14d6-adf5-48b8-b1ce-4bbe04f2e0bd, queue=awx_private_queue)
[2021-04-08 11:06:53,828: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ecdb14d6-adf5-48b8-b1ce-4bbe04f2e0bd, queue=awx_private_queue)
2021-04-08 11:06:53,870 DEBUG awx.main.dispatch delivered ecdb14d6-adf5-48b8-b1ce-4bbe04f2e0bd to worker[178] qsize 0
2021-04-08 11:06:53,872 DEBUG awx.main.dispatch task ecdb14d6-adf5-48b8-b1ce-4bbe04f2e0bd starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:06:53,881 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:06:53,885 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:06:23.870372+00:00
2021-04-08 11:06:53,896 DEBUG awx.main.dispatch task ecdb14d6-adf5-48b8-b1ce-4bbe04f2e0bd is finished
2021-04-08 11:06:54,046 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cbe5e409-87cb-4e7c-b7fa-696035325167, queue=awx_private_queue)
[2021-04-08 11:06:54,046: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cbe5e409-87cb-4e7c-b7fa-696035325167, queue=awx_private_queue)
2021-04-08 11:06:54,218 DEBUG awx.main.dispatch delivered cbe5e409-87cb-4e7c-b7fa-696035325167 to worker[179] qsize 0
2021-04-08 11:06:54,225 DEBUG awx.main.dispatch task cbe5e409-87cb-4e7c-b7fa-696035325167 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:06:54,235 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:06:54,260 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:06:54,292 DEBUG awx.main.dispatch task cbe5e409-87cb-4e7c-b7fa-696035325167 is finished
RESULT 2
OKREADY
2021-04-08 11:07:14,054 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d4506d61-228c-4dbb-b3ee-77e183d20ff0, queue=awx_private_queue)
[2021-04-08 11:07:14,054: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d4506d61-228c-4dbb-b3ee-77e183d20ff0, queue=awx_private_queue)
2021-04-08 11:07:14,074 DEBUG awx.main.dispatch delivered d4506d61-228c-4dbb-b3ee-77e183d20ff0 to worker[176] qsize 0
2021-04-08 11:07:14,077 DEBUG awx.main.dispatch task d4506d61-228c-4dbb-b3ee-77e183d20ff0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:07:14,079 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:07:14,093 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:07:14,117 DEBUG awx.main.dispatch task d4506d61-228c-4dbb-b3ee-77e183d20ff0 is finished
2021-04-08 11:07:23,838 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(eb655ab3-4437-49c0-8472-eaf87f86db59, queue=awx_private_queue)
[2021-04-08 11:07:23,838: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(eb655ab3-4437-49c0-8472-eaf87f86db59, queue=awx_private_queue)
2021-04-08 11:07:23,864 DEBUG awx.main.dispatch delivered eb655ab3-4437-49c0-8472-eaf87f86db59 to worker[179] qsize 0
2021-04-08 11:07:23,871 DEBUG awx.main.dispatch task eb655ab3-4437-49c0-8472-eaf87f86db59 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:07:23,884 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:07:23,888 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:06:53.881746+00:00
2021-04-08 11:07:23,903 DEBUG awx.main.dispatch task eb655ab3-4437-49c0-8472-eaf87f86db59 is finished
2021-04-08 11:07:34,052 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(00370ef9-24f3-4665-99fc-81e92bdb2147, queue=awx_private_queue)
[2021-04-08 11:07:34,052: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(00370ef9-24f3-4665-99fc-81e92bdb2147, queue=awx_private_queue)
2021-04-08 11:07:34,066 DEBUG awx.main.dispatch delivered 00370ef9-24f3-4665-99fc-81e92bdb2147 to worker[176] qsize 0
2021-04-08 11:07:34,069 DEBUG awx.main.dispatch task 00370ef9-24f3-4665-99fc-81e92bdb2147 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:07:34,072 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:07:34,089 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:07:34,118 DEBUG awx.main.dispatch task 00370ef9-24f3-4665-99fc-81e92bdb2147 is finished
2021-04-08 11:07:53,447 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(840070fe-101c-474e-8095-7b4765f31c36, queue=dev-awx-0)
[2021-04-08 11:07:53,447: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(840070fe-101c-474e-8095-7b4765f31c36, queue=dev-awx-0)
2021-04-08 11:07:53,485 DEBUG awx.main.dispatch delivered 840070fe-101c-474e-8095-7b4765f31c36 to worker[178] qsize 0
2021-04-08 11:07:53,491 DEBUG awx.main.dispatch task 840070fe-101c-474e-8095-7b4765f31c36 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:07:53,498 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:07:53,530 DEBUG awx.main.dispatch task 840070fe-101c-474e-8095-7b4765f31c36 is finished
2021-04-08 11:07:53,619 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(62ddbeae-255e-4caf-9840-087ceea7a40a, queue=dev-awx-0)
[2021-04-08 11:07:53,619: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(62ddbeae-255e-4caf-9840-087ceea7a40a, queue=dev-awx-0)
2021-04-08 11:07:53,632 DEBUG awx.main.dispatch delivered 62ddbeae-255e-4caf-9840-087ceea7a40a to worker[177] qsize 0
2021-04-08 11:07:53,637 DEBUG awx.main.dispatch task 62ddbeae-255e-4caf-9840-087ceea7a40a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:07:53,660 DEBUG awx.main.dispatch task 62ddbeae-255e-4caf-9840-087ceea7a40a is finished
2021-04-08 11:07:53,830 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4ef8fccb-4f89-4e55-b6b1-8d8284ab92b7, queue=awx_private_queue)
[2021-04-08 11:07:53,830: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4ef8fccb-4f89-4e55-b6b1-8d8284ab92b7, queue=awx_private_queue)
2021-04-08 11:07:53,861 DEBUG awx.main.dispatch delivered 4ef8fccb-4f89-4e55-b6b1-8d8284ab92b7 to worker[179] qsize 0
2021-04-08 11:07:53,863 DEBUG awx.main.dispatch task 4ef8fccb-4f89-4e55-b6b1-8d8284ab92b7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:07:53,871 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:07:53,873 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:07:23.884693+00:00
2021-04-08 11:07:53,882 DEBUG awx.main.dispatch task 4ef8fccb-4f89-4e55-b6b1-8d8284ab92b7 is finished
2021-04-08 11:07:54,052 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(45422eca-da1c-430a-8f94-792f45e91ddb, queue=awx_private_queue)
[2021-04-08 11:07:54,052: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(45422eca-da1c-430a-8f94-792f45e91ddb, queue=awx_private_queue)
2021-04-08 11:07:54,243 DEBUG awx.main.dispatch delivered 45422eca-da1c-430a-8f94-792f45e91ddb to worker[178] qsize 0
2021-04-08 11:07:54,247 DEBUG awx.main.dispatch task 45422eca-da1c-430a-8f94-792f45e91ddb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:07:54,248 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:07:54,263 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:07:54,294 DEBUG awx.main.dispatch task 45422eca-da1c-430a-8f94-792f45e91ddb is finished
RESULT 2
OKREADY
2021-04-08 11:08:14,064 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6494791d-bebf-422b-bfe0-19446ca34648, queue=awx_private_queue)
[2021-04-08 11:08:14,064: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6494791d-bebf-422b-bfe0-19446ca34648, queue=awx_private_queue)
2021-04-08 11:08:14,078 DEBUG awx.main.dispatch delivered 6494791d-bebf-422b-bfe0-19446ca34648 to worker[178] qsize 0
2021-04-08 11:08:14,081 DEBUG awx.main.dispatch task 6494791d-bebf-422b-bfe0-19446ca34648 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:08:14,082 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:08:14,094 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:08:14,119 DEBUG awx.main.dispatch task 6494791d-bebf-422b-bfe0-19446ca34648 is finished
2021-04-08 11:08:23,830 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9cd40a35-4229-4c7a-9b72-dd61d59d8fe4, queue=awx_private_queue)
[2021-04-08 11:08:23,830: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9cd40a35-4229-4c7a-9b72-dd61d59d8fe4, queue=awx_private_queue)
2021-04-08 11:08:23,845 DEBUG awx.main.dispatch delivered 9cd40a35-4229-4c7a-9b72-dd61d59d8fe4 to worker[177] qsize 0
2021-04-08 11:08:23,847 DEBUG awx.main.dispatch task 9cd40a35-4229-4c7a-9b72-dd61d59d8fe4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:08:23,857 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:08:23,861 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:07:53.871382+00:00
2021-04-08 11:08:23,877 DEBUG awx.main.dispatch task 9cd40a35-4229-4c7a-9b72-dd61d59d8fe4 is finished
2021-04-08 11:08:34,064 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(40df91c9-5d92-439c-b526-e35b958fbf83, queue=awx_private_queue)
[2021-04-08 11:08:34,064: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(40df91c9-5d92-439c-b526-e35b958fbf83, queue=awx_private_queue)
2021-04-08 11:08:34,079 DEBUG awx.main.dispatch delivered 40df91c9-5d92-439c-b526-e35b958fbf83 to worker[176] qsize 0
2021-04-08 11:08:34,081 DEBUG awx.main.dispatch task 40df91c9-5d92-439c-b526-e35b958fbf83 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:08:34,085 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:08:34,102 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:08:34,134 DEBUG awx.main.dispatch task 40df91c9-5d92-439c-b526-e35b958fbf83 is finished
2021-04-08 11:08:53,458 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c4a77034-8db1-42e8-b08d-367a9ab6d1dc, queue=dev-awx-0)
[2021-04-08 11:08:53,458: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c4a77034-8db1-42e8-b08d-367a9ab6d1dc, queue=dev-awx-0)
2021-04-08 11:08:53,497 DEBUG awx.main.dispatch delivered c4a77034-8db1-42e8-b08d-367a9ab6d1dc to worker[179] qsize 0
2021-04-08 11:08:53,502 DEBUG awx.main.dispatch task c4a77034-8db1-42e8-b08d-367a9ab6d1dc starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:08:53,506 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:08:53,532 DEBUG awx.main.dispatch task c4a77034-8db1-42e8-b08d-367a9ab6d1dc is finished
2021-04-08 11:08:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(fb5925e1-49a4-481f-a229-643b29653b8e, queue=dev-awx-0)
[2021-04-08 11:08:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(fb5925e1-49a4-481f-a229-643b29653b8e, queue=dev-awx-0)
2021-04-08 11:08:53,634 DEBUG awx.main.dispatch delivered fb5925e1-49a4-481f-a229-643b29653b8e to worker[179] qsize 0
2021-04-08 11:08:53,636 DEBUG awx.main.dispatch task fb5925e1-49a4-481f-a229-643b29653b8e starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:08:53,652 DEBUG awx.main.dispatch task fb5925e1-49a4-481f-a229-643b29653b8e is finished
2021-04-08 11:08:53,830 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c59850e3-a8b3-4ccd-8c12-403c2ff427c2, queue=awx_private_queue)
[2021-04-08 11:08:53,830: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c59850e3-a8b3-4ccd-8c12-403c2ff427c2, queue=awx_private_queue)
2021-04-08 11:08:53,844 DEBUG awx.main.dispatch delivered c59850e3-a8b3-4ccd-8c12-403c2ff427c2 to worker[176] qsize 0
2021-04-08 11:08:53,851 DEBUG awx.main.dispatch task c59850e3-a8b3-4ccd-8c12-403c2ff427c2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:08:53,862 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:08:53,867 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:08:23.857959+00:00
2021-04-08 11:08:53,881 DEBUG awx.main.dispatch task c59850e3-a8b3-4ccd-8c12-403c2ff427c2 is finished
2021-04-08 11:08:54,064 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0434c2bc-22c3-4f2e-8898-fa6a4df05770, queue=awx_private_queue)
[2021-04-08 11:08:54,064: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0434c2bc-22c3-4f2e-8898-fa6a4df05770, queue=awx_private_queue)
2021-04-08 11:08:54,264 DEBUG awx.main.dispatch delivered 0434c2bc-22c3-4f2e-8898-fa6a4df05770 to worker[179] qsize 0
2021-04-08 11:08:54,266 DEBUG awx.main.dispatch task 0434c2bc-22c3-4f2e-8898-fa6a4df05770 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:08:54,267 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:08:54,290 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:08:54,332 DEBUG awx.main.dispatch task 0434c2bc-22c3-4f2e-8898-fa6a4df05770 is finished
RESULT 2
OKREADY
2021-04-08 11:09:14,075 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b2ebd0d8-e31e-43b8-88c8-5b074c3e51df, queue=awx_private_queue)
[2021-04-08 11:09:14,075: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b2ebd0d8-e31e-43b8-88c8-5b074c3e51df, queue=awx_private_queue)
2021-04-08 11:09:14,092 DEBUG awx.main.dispatch delivered b2ebd0d8-e31e-43b8-88c8-5b074c3e51df to worker[176] qsize 0
2021-04-08 11:09:14,094 DEBUG awx.main.dispatch task b2ebd0d8-e31e-43b8-88c8-5b074c3e51df starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:09:14,096 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:09:14,111 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:09:14,135 DEBUG awx.main.dispatch task b2ebd0d8-e31e-43b8-88c8-5b074c3e51df is finished
2021-04-08 11:09:23,830 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e227522f-9f90-4a8c-b7c5-0ebe7d6ffb7a, queue=awx_private_queue)
[2021-04-08 11:09:23,830: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e227522f-9f90-4a8c-b7c5-0ebe7d6ffb7a, queue=awx_private_queue)
2021-04-08 11:09:23,843 DEBUG awx.main.dispatch delivered e227522f-9f90-4a8c-b7c5-0ebe7d6ffb7a to worker[179] qsize 0
2021-04-08 11:09:23,847 DEBUG awx.main.dispatch task e227522f-9f90-4a8c-b7c5-0ebe7d6ffb7a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:09:23,857 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:09:23,860 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:08:53.862383+00:00
2021-04-08 11:09:23,873 DEBUG awx.main.dispatch task e227522f-9f90-4a8c-b7c5-0ebe7d6ffb7a is finished
2021-04-08 11:09:34,073 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f6c07c4f-3ad4-4d13-aedf-fd120facfa34, queue=awx_private_queue)
[2021-04-08 11:09:34,073: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f6c07c4f-3ad4-4d13-aedf-fd120facfa34, queue=awx_private_queue)
2021-04-08 11:09:34,101 DEBUG awx.main.dispatch delivered f6c07c4f-3ad4-4d13-aedf-fd120facfa34 to worker[177] qsize 0
2021-04-08 11:09:34,108 DEBUG awx.main.dispatch task f6c07c4f-3ad4-4d13-aedf-fd120facfa34 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:09:34,117 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:09:34,141 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:09:34,164 DEBUG awx.main.dispatch task f6c07c4f-3ad4-4d13-aedf-fd120facfa34 is finished
2021-04-08 11:09:53,464 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6bf2b5ac-ea2a-41a6-bcb7-f09a9ba52491, queue=dev-awx-0)
[2021-04-08 11:09:53,464: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6bf2b5ac-ea2a-41a6-bcb7-f09a9ba52491, queue=dev-awx-0)
2021-04-08 11:09:53,493 DEBUG awx.main.dispatch delivered 6bf2b5ac-ea2a-41a6-bcb7-f09a9ba52491 to worker[177] qsize 0
2021-04-08 11:09:53,496 DEBUG awx.main.dispatch task 6bf2b5ac-ea2a-41a6-bcb7-f09a9ba52491 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:09:53,498 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:09:53,525 DEBUG awx.main.dispatch task 6bf2b5ac-ea2a-41a6-bcb7-f09a9ba52491 is finished
2021-04-08 11:09:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(cf12f994-0b87-4ac9-b8d0-fd1409052809, queue=dev-awx-0)
[2021-04-08 11:09:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(cf12f994-0b87-4ac9-b8d0-fd1409052809, queue=dev-awx-0)
2021-04-08 11:09:53,631 DEBUG awx.main.dispatch delivered cf12f994-0b87-4ac9-b8d0-fd1409052809 to worker[177] qsize 0
2021-04-08 11:09:53,634 DEBUG awx.main.dispatch task cf12f994-0b87-4ac9-b8d0-fd1409052809 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:09:53,669 DEBUG awx.main.dispatch task cf12f994-0b87-4ac9-b8d0-fd1409052809 is finished
2021-04-08 11:09:53,830 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ed616904-2b94-428d-9987-1501d8ff4980, queue=awx_private_queue)
[2021-04-08 11:09:53,830: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ed616904-2b94-428d-9987-1501d8ff4980, queue=awx_private_queue)
2021-04-08 11:09:53,844 DEBUG awx.main.dispatch delivered ed616904-2b94-428d-9987-1501d8ff4980 to worker[177] qsize 0
2021-04-08 11:09:53,846 DEBUG awx.main.dispatch task ed616904-2b94-428d-9987-1501d8ff4980 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:09:53,853 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:09:53,856 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:09:23.857986+00:00
2021-04-08 11:09:53,868 DEBUG awx.main.dispatch task ed616904-2b94-428d-9987-1501d8ff4980 is finished
2021-04-08 11:09:54,073 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3bdac2f3-07ef-4b5c-91c3-a033376ef744, queue=awx_private_queue)
[2021-04-08 11:09:54,073: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3bdac2f3-07ef-4b5c-91c3-a033376ef744, queue=awx_private_queue)
2021-04-08 11:09:54,246 DEBUG awx.main.dispatch delivered 3bdac2f3-07ef-4b5c-91c3-a033376ef744 to worker[177] qsize 0
2021-04-08 11:09:54,249 DEBUG awx.main.dispatch task 3bdac2f3-07ef-4b5c-91c3-a033376ef744 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:09:54,250 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:09:54,264 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:09:54,295 DEBUG awx.main.dispatch task 3bdac2f3-07ef-4b5c-91c3-a033376ef744 is finished
RESULT 2
OKREADY
2021-04-08 11:10:14,090 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cea10940-2798-4972-8024-9d7e0c04452b, queue=awx_private_queue)
[2021-04-08 11:10:14,090: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cea10940-2798-4972-8024-9d7e0c04452b, queue=awx_private_queue)
2021-04-08 11:10:14,114 DEBUG awx.main.dispatch delivered cea10940-2798-4972-8024-9d7e0c04452b to worker[176] qsize 0
2021-04-08 11:10:14,123 DEBUG awx.main.dispatch task cea10940-2798-4972-8024-9d7e0c04452b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:10:14,130 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:10:14,156 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:10:14,183 DEBUG awx.main.dispatch task cea10940-2798-4972-8024-9d7e0c04452b is finished
2021-04-08 11:10:23,832 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d37cf5ac-d54d-4d11-9e95-79f6b94b3432, queue=awx_private_queue)
[2021-04-08 11:10:23,832: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d37cf5ac-d54d-4d11-9e95-79f6b94b3432, queue=awx_private_queue)
2021-04-08 11:10:23,860 DEBUG awx.main.dispatch delivered d37cf5ac-d54d-4d11-9e95-79f6b94b3432 to worker[176] qsize 0
2021-04-08 11:10:23,862 DEBUG awx.main.dispatch task d37cf5ac-d54d-4d11-9e95-79f6b94b3432 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:10:23,872 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:10:23,876 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:09:53.853738+00:00
2021-04-08 11:10:23,890 DEBUG awx.main.dispatch task d37cf5ac-d54d-4d11-9e95-79f6b94b3432 is finished
2021-04-08 11:10:34,087 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6d9bd395-c5c1-45a8-b705-c54d8cc872c0, queue=awx_private_queue)
[2021-04-08 11:10:34,087: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6d9bd395-c5c1-45a8-b705-c54d8cc872c0, queue=awx_private_queue)
2021-04-08 11:10:34,110 DEBUG awx.main.dispatch delivered 6d9bd395-c5c1-45a8-b705-c54d8cc872c0 to worker[176] qsize 0
2021-04-08 11:10:34,112 DEBUG awx.main.dispatch task 6d9bd395-c5c1-45a8-b705-c54d8cc872c0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:10:34,114 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:10:34,133 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:10:34,171 DEBUG awx.main.dispatch task 6d9bd395-c5c1-45a8-b705-c54d8cc872c0 is finished
2021-04-08 11:10:53,465 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(511e39ee-04b9-4400-abe1-d1431d2094ad, queue=dev-awx-0)
[2021-04-08 11:10:53,465: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(511e39ee-04b9-4400-abe1-d1431d2094ad, queue=dev-awx-0)
2021-04-08 11:10:53,501 DEBUG awx.main.dispatch delivered 511e39ee-04b9-4400-abe1-d1431d2094ad to worker[176] qsize 0
2021-04-08 11:10:53,504 DEBUG awx.main.dispatch task 511e39ee-04b9-4400-abe1-d1431d2094ad starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:10:53,506 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:10:53,526 DEBUG awx.main.dispatch task 511e39ee-04b9-4400-abe1-d1431d2094ad is finished
2021-04-08 11:10:53,618 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(fba69295-822c-4d82-8dc6-69f91074f2cd, queue=dev-awx-0)
[2021-04-08 11:10:53,618: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(fba69295-822c-4d82-8dc6-69f91074f2cd, queue=dev-awx-0)
2021-04-08 11:10:53,634 DEBUG awx.main.dispatch delivered fba69295-822c-4d82-8dc6-69f91074f2cd to worker[176] qsize 0
2021-04-08 11:10:53,643 DEBUG awx.main.dispatch task fba69295-822c-4d82-8dc6-69f91074f2cd starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:10:53,663 DEBUG awx.main.dispatch task fba69295-822c-4d82-8dc6-69f91074f2cd is finished
2021-04-08 11:10:53,831 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a8a5c3b8-a25f-43c2-8c3d-0cc9bf4f294b, queue=awx_private_queue)
[2021-04-08 11:10:53,831: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a8a5c3b8-a25f-43c2-8c3d-0cc9bf4f294b, queue=awx_private_queue)
2021-04-08 11:10:53,853 DEBUG awx.main.dispatch delivered a8a5c3b8-a25f-43c2-8c3d-0cc9bf4f294b to worker[178] qsize 0
2021-04-08 11:10:53,857 DEBUG awx.main.dispatch task a8a5c3b8-a25f-43c2-8c3d-0cc9bf4f294b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:10:53,870 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:10:53,875 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:10:23.873091+00:00
2021-04-08 11:10:53,888 DEBUG awx.main.dispatch task a8a5c3b8-a25f-43c2-8c3d-0cc9bf4f294b is finished
2021-04-08 11:10:54,086 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0237443e-c4d4-45f4-acbd-2a6ce880ff8f, queue=awx_private_queue)
[2021-04-08 11:10:54,086: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0237443e-c4d4-45f4-acbd-2a6ce880ff8f, queue=awx_private_queue)
2021-04-08 11:10:54,279 DEBUG awx.main.dispatch delivered 0237443e-c4d4-45f4-acbd-2a6ce880ff8f to worker[176] qsize 0
2021-04-08 11:10:54,281 DEBUG awx.main.dispatch task 0237443e-c4d4-45f4-acbd-2a6ce880ff8f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:10:54,282 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:10:54,301 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:10:54,332 DEBUG awx.main.dispatch task 0237443e-c4d4-45f4-acbd-2a6ce880ff8f is finished
RESULT 2
OKREADY
2021-04-08 11:11:14,099 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e38c4e41-5d1d-4f8d-8f96-68a71cedca14, queue=awx_private_queue)
[2021-04-08 11:11:14,099: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e38c4e41-5d1d-4f8d-8f96-68a71cedca14, queue=awx_private_queue)
2021-04-08 11:11:14,113 DEBUG awx.main.dispatch delivered e38c4e41-5d1d-4f8d-8f96-68a71cedca14 to worker[179] qsize 0
2021-04-08 11:11:14,116 DEBUG awx.main.dispatch task e38c4e41-5d1d-4f8d-8f96-68a71cedca14 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:11:14,121 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:11:14,140 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:11:14,172 DEBUG awx.main.dispatch task e38c4e41-5d1d-4f8d-8f96-68a71cedca14 is finished
2021-04-08 11:11:23,831 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f70e69d9-facc-4c08-8230-daf9813ebe60, queue=awx_private_queue)
[2021-04-08 11:11:23,831: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f70e69d9-facc-4c08-8230-daf9813ebe60, queue=awx_private_queue)
2021-04-08 11:11:23,844 DEBUG awx.main.dispatch delivered f70e69d9-facc-4c08-8230-daf9813ebe60 to worker[179] qsize 0
2021-04-08 11:11:23,846 DEBUG awx.main.dispatch task f70e69d9-facc-4c08-8230-daf9813ebe60 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:11:23,859 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:11:23,862 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:10:53.870951+00:00
2021-04-08 11:11:23,873 DEBUG awx.main.dispatch task f70e69d9-facc-4c08-8230-daf9813ebe60 is finished
2021-04-08 11:11:34,102 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(657533d4-e2f8-483f-9c06-26afd4ba9074, queue=awx_private_queue)
[2021-04-08 11:11:34,102: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(657533d4-e2f8-483f-9c06-26afd4ba9074, queue=awx_private_queue)
2021-04-08 11:11:34,125 DEBUG awx.main.dispatch delivered 657533d4-e2f8-483f-9c06-26afd4ba9074 to worker[179] qsize 0
2021-04-08 11:11:34,133 DEBUG awx.main.dispatch task 657533d4-e2f8-483f-9c06-26afd4ba9074 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:11:34,138 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:11:34,170 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:11:34,198 DEBUG awx.main.dispatch task 657533d4-e2f8-483f-9c06-26afd4ba9074 is finished
2021-04-08 11:11:53,476 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(cd8062df-7ef6-4509-85d3-fa3737a56a59, queue=dev-awx-0)
[2021-04-08 11:11:53,476: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(cd8062df-7ef6-4509-85d3-fa3737a56a59, queue=dev-awx-0)
2021-04-08 11:11:53,516 DEBUG awx.main.dispatch delivered cd8062df-7ef6-4509-85d3-fa3737a56a59 to worker[177] qsize 0
2021-04-08 11:11:53,521 DEBUG awx.main.dispatch task cd8062df-7ef6-4509-85d3-fa3737a56a59 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:11:53,527 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:11:53,557 DEBUG awx.main.dispatch task cd8062df-7ef6-4509-85d3-fa3737a56a59 is finished
2021-04-08 11:11:53,619 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(86c2ba3f-18b5-4bf5-87bc-0afbbb6fdb08, queue=dev-awx-0)
[2021-04-08 11:11:53,619: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(86c2ba3f-18b5-4bf5-87bc-0afbbb6fdb08, queue=dev-awx-0)
2021-04-08 11:11:53,644 DEBUG awx.main.dispatch delivered 86c2ba3f-18b5-4bf5-87bc-0afbbb6fdb08 to worker[176] qsize 0
2021-04-08 11:11:53,648 DEBUG awx.main.dispatch task 86c2ba3f-18b5-4bf5-87bc-0afbbb6fdb08 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:11:53,664 DEBUG awx.main.dispatch task 86c2ba3f-18b5-4bf5-87bc-0afbbb6fdb08 is finished
2021-04-08 11:11:53,831 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4cb0ded5-b56f-4716-b9b6-9f399eb1ecb0, queue=awx_private_queue)
[2021-04-08 11:11:53,831: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4cb0ded5-b56f-4716-b9b6-9f399eb1ecb0, queue=awx_private_queue)
2021-04-08 11:11:53,852 DEBUG awx.main.dispatch delivered 4cb0ded5-b56f-4716-b9b6-9f399eb1ecb0 to worker[176] qsize 0
2021-04-08 11:11:53,854 DEBUG awx.main.dispatch task 4cb0ded5-b56f-4716-b9b6-9f399eb1ecb0 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:11:53,861 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:11:53,864 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:11:23.860055+00:00
2021-04-08 11:11:53,877 DEBUG awx.main.dispatch task 4cb0ded5-b56f-4716-b9b6-9f399eb1ecb0 is finished
2021-04-08 11:11:54,099 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(764ead1b-4a3f-4c7b-a6d9-d9f6ccf601ff, queue=awx_private_queue)
[2021-04-08 11:11:54,099: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(764ead1b-4a3f-4c7b-a6d9-d9f6ccf601ff, queue=awx_private_queue)
2021-04-08 11:11:54,299 DEBUG awx.main.dispatch delivered 764ead1b-4a3f-4c7b-a6d9-d9f6ccf601ff to worker[178] qsize 0
2021-04-08 11:11:54,308 DEBUG awx.main.dispatch task 764ead1b-4a3f-4c7b-a6d9-d9f6ccf601ff starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:11:54,319 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:11:54,348 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:11:54,374 DEBUG awx.main.dispatch task 764ead1b-4a3f-4c7b-a6d9-d9f6ccf601ff is finished
RESULT 2
OKREADY
2021-04-08 11:12:14,106 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1966114b-f8dc-4973-b70c-b18e60dfd6fd, queue=awx_private_queue)
[2021-04-08 11:12:14,106: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1966114b-f8dc-4973-b70c-b18e60dfd6fd, queue=awx_private_queue)
2021-04-08 11:12:14,128 DEBUG awx.main.dispatch delivered 1966114b-f8dc-4973-b70c-b18e60dfd6fd to worker[176] qsize 0
2021-04-08 11:12:14,132 DEBUG awx.main.dispatch task 1966114b-f8dc-4973-b70c-b18e60dfd6fd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:12:14,133 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:12:14,150 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:12:14,173 DEBUG awx.main.dispatch task 1966114b-f8dc-4973-b70c-b18e60dfd6fd is finished
2021-04-08 11:12:23,831 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(76583fc5-5d3a-4574-9f6c-3456f3bae387, queue=awx_private_queue)
[2021-04-08 11:12:23,831: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(76583fc5-5d3a-4574-9f6c-3456f3bae387, queue=awx_private_queue)
2021-04-08 11:12:23,844 DEBUG awx.main.dispatch delivered 76583fc5-5d3a-4574-9f6c-3456f3bae387 to worker[176] qsize 0
2021-04-08 11:12:23,846 DEBUG awx.main.dispatch task 76583fc5-5d3a-4574-9f6c-3456f3bae387 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:12:23,857 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:12:23,859 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:11:53.861829+00:00
2021-04-08 11:12:23,870 DEBUG awx.main.dispatch task 76583fc5-5d3a-4574-9f6c-3456f3bae387 is finished
2021-04-08 11:12:34,105 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(89488438-948b-40de-bb87-b2c8880187b8, queue=awx_private_queue)
[2021-04-08 11:12:34,105: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(89488438-948b-40de-bb87-b2c8880187b8, queue=awx_private_queue)
2021-04-08 11:12:34,126 DEBUG awx.main.dispatch delivered 89488438-948b-40de-bb87-b2c8880187b8 to worker[177] qsize 0
2021-04-08 11:12:34,129 DEBUG awx.main.dispatch task 89488438-948b-40de-bb87-b2c8880187b8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:12:34,131 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:12:34,147 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:12:34,172 DEBUG awx.main.dispatch task 89488438-948b-40de-bb87-b2c8880187b8 is finished
2021-04-08 11:12:53,489 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(4848cdf7-5472-4585-9bd9-dc76ae40c7d8, queue=dev-awx-0)
[2021-04-08 11:12:53,489: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(4848cdf7-5472-4585-9bd9-dc76ae40c7d8, queue=dev-awx-0)
2021-04-08 11:12:53,526 DEBUG awx.main.dispatch delivered 4848cdf7-5472-4585-9bd9-dc76ae40c7d8 to worker[177] qsize 0
2021-04-08 11:12:53,530 DEBUG awx.main.dispatch task 4848cdf7-5472-4585-9bd9-dc76ae40c7d8 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:12:53,534 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:12:53,554 DEBUG awx.main.dispatch task 4848cdf7-5472-4585-9bd9-dc76ae40c7d8 is finished
2021-04-08 11:12:53,619 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2e964291-c4dc-48f5-9232-5a115f801de2, queue=dev-awx-0)
[2021-04-08 11:12:53,619: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2e964291-c4dc-48f5-9232-5a115f801de2, queue=dev-awx-0)
2021-04-08 11:12:53,633 DEBUG awx.main.dispatch delivered 2e964291-c4dc-48f5-9232-5a115f801de2 to worker[179] qsize 0
2021-04-08 11:12:53,638 DEBUG awx.main.dispatch task 2e964291-c4dc-48f5-9232-5a115f801de2 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:12:53,662 DEBUG awx.main.dispatch task 2e964291-c4dc-48f5-9232-5a115f801de2 is finished
2021-04-08 11:12:53,831 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(00fe6649-e8a6-4250-baec-90b115271a49, queue=awx_private_queue)
[2021-04-08 11:12:53,831: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(00fe6649-e8a6-4250-baec-90b115271a49, queue=awx_private_queue)
2021-04-08 11:12:53,846 DEBUG awx.main.dispatch delivered 00fe6649-e8a6-4250-baec-90b115271a49 to worker[177] qsize 0
2021-04-08 11:12:53,849 DEBUG awx.main.dispatch task 00fe6649-e8a6-4250-baec-90b115271a49 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:12:53,857 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:12:53,859 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:12:23.857517+00:00
2021-04-08 11:12:53,871 DEBUG awx.main.dispatch task 00fe6649-e8a6-4250-baec-90b115271a49 is finished
2021-04-08 11:12:54,105 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8286b8a0-69aa-4064-8aa9-261ac0ec0a39, queue=awx_private_queue)
[2021-04-08 11:12:54,105: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8286b8a0-69aa-4064-8aa9-261ac0ec0a39, queue=awx_private_queue)
2021-04-08 11:12:54,120 DEBUG awx.main.dispatch delivered 8286b8a0-69aa-4064-8aa9-261ac0ec0a39 to worker[177] qsize 0
2021-04-08 11:12:54,133 DEBUG awx.main.dispatch task 8286b8a0-69aa-4064-8aa9-261ac0ec0a39 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:12:54,135 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:12:54,147 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:12:54,173 DEBUG awx.main.dispatch task 8286b8a0-69aa-4064-8aa9-261ac0ec0a39 is finished
RESULT 2
OKREADY
2021-04-08 11:13:14,116 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0ece003e-3852-4a79-af1a-ef9688830908, queue=awx_private_queue)
[2021-04-08 11:13:14,116: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0ece003e-3852-4a79-af1a-ef9688830908, queue=awx_private_queue)
2021-04-08 11:13:14,137 DEBUG awx.main.dispatch delivered 0ece003e-3852-4a79-af1a-ef9688830908 to worker[176] qsize 0
2021-04-08 11:13:14,141 DEBUG awx.main.dispatch task 0ece003e-3852-4a79-af1a-ef9688830908 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:13:14,328 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:13:14,336 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:13:14,379 DEBUG awx.main.dispatch task 0ece003e-3852-4a79-af1a-ef9688830908 is finished
2021-04-08 11:13:23,833 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(386031c3-bcbf-4579-ba23-b014403458a5, queue=awx_private_queue)
[2021-04-08 11:13:23,833: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(386031c3-bcbf-4579-ba23-b014403458a5, queue=awx_private_queue)
2021-04-08 11:13:23,868 DEBUG awx.main.dispatch delivered 386031c3-bcbf-4579-ba23-b014403458a5 to worker[179] qsize 0
2021-04-08 11:13:23,871 DEBUG awx.main.dispatch task 386031c3-bcbf-4579-ba23-b014403458a5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:13:23,884 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:13:23,890 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:12:53.857948+00:00
2021-04-08 11:13:23,907 DEBUG awx.main.dispatch task 386031c3-bcbf-4579-ba23-b014403458a5 is finished
2021-04-08 11:13:34,114 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7f3cc6e4-50a1-44d5-bac2-67ca249d6fd2, queue=awx_private_queue)
[2021-04-08 11:13:34,114: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7f3cc6e4-50a1-44d5-bac2-67ca249d6fd2, queue=awx_private_queue)
2021-04-08 11:13:34,137 DEBUG awx.main.dispatch delivered 7f3cc6e4-50a1-44d5-bac2-67ca249d6fd2 to worker[179] qsize 0
2021-04-08 11:13:34,139 DEBUG awx.main.dispatch task 7f3cc6e4-50a1-44d5-bac2-67ca249d6fd2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:13:34,140 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:13:34,155 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:13:34,180 DEBUG awx.main.dispatch task 7f3cc6e4-50a1-44d5-bac2-67ca249d6fd2 is finished
2021-04-08 11:13:53,485 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(86f33e14-bf22-4c6b-885e-ab926947468f, queue=dev-awx-0)
[2021-04-08 11:13:53,485: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(86f33e14-bf22-4c6b-885e-ab926947468f, queue=dev-awx-0)
2021-04-08 11:13:53,517 DEBUG awx.main.dispatch delivered 86f33e14-bf22-4c6b-885e-ab926947468f to worker[177] qsize 0
2021-04-08 11:13:53,522 DEBUG awx.main.dispatch task 86f33e14-bf22-4c6b-885e-ab926947468f starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:13:53,525 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:13:53,554 DEBUG awx.main.dispatch task 86f33e14-bf22-4c6b-885e-ab926947468f is finished
2021-04-08 11:13:53,619 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(94f547c4-d5a7-4e47-8201-d26d24f18185, queue=dev-awx-0)
[2021-04-08 11:13:53,619: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(94f547c4-d5a7-4e47-8201-d26d24f18185, queue=dev-awx-0)
2021-04-08 11:13:53,635 DEBUG awx.main.dispatch delivered 94f547c4-d5a7-4e47-8201-d26d24f18185 to worker[179] qsize 0
2021-04-08 11:13:53,637 DEBUG awx.main.dispatch task 94f547c4-d5a7-4e47-8201-d26d24f18185 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:13:53,647 DEBUG awx.main.dispatch task 94f547c4-d5a7-4e47-8201-d26d24f18185 is finished
2021-04-08 11:13:53,831 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ab5ddc60-b70f-44db-9760-f6724ce4c975, queue=awx_private_queue)
[2021-04-08 11:13:53,831: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ab5ddc60-b70f-44db-9760-f6724ce4c975, queue=awx_private_queue)
2021-04-08 11:13:53,847 DEBUG awx.main.dispatch delivered ab5ddc60-b70f-44db-9760-f6724ce4c975 to worker[176] qsize 0
2021-04-08 11:13:53,849 DEBUG awx.main.dispatch task ab5ddc60-b70f-44db-9760-f6724ce4c975 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:13:53,858 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:13:53,860 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:13:23.884365+00:00
2021-04-08 11:13:53,869 DEBUG awx.main.dispatch task ab5ddc60-b70f-44db-9760-f6724ce4c975 is finished
2021-04-08 11:13:54,114 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0479470c-e50b-4057-9534-0944fbf44c8d, queue=awx_private_queue)
[2021-04-08 11:13:54,114: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0479470c-e50b-4057-9534-0944fbf44c8d, queue=awx_private_queue)
2021-04-08 11:13:54,128 DEBUG awx.main.dispatch delivered 0479470c-e50b-4057-9534-0944fbf44c8d to worker[178] qsize 0
2021-04-08 11:13:54,132 DEBUG awx.main.dispatch task 0479470c-e50b-4057-9534-0944fbf44c8d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:13:54,136 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:13:54,155 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:13:54,178 DEBUG awx.main.dispatch task 0479470c-e50b-4057-9534-0944fbf44c8d is finished
RESULT 2
OKREADY
2021-04-08 11:14:14,135 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(872fca0b-3b4e-485a-bb2c-90c1abc4f46a, queue=awx_private_queue)
[2021-04-08 11:14:14,135: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(872fca0b-3b4e-485a-bb2c-90c1abc4f46a, queue=awx_private_queue)
2021-04-08 11:14:14,163 DEBUG awx.main.dispatch delivered 872fca0b-3b4e-485a-bb2c-90c1abc4f46a to worker[177] qsize 0
2021-04-08 11:14:14,174 DEBUG awx.main.dispatch task 872fca0b-3b4e-485a-bb2c-90c1abc4f46a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:14:14,183 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:14:14,209 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:14:14,239 DEBUG awx.main.dispatch task 872fca0b-3b4e-485a-bb2c-90c1abc4f46a is finished
2021-04-08 11:14:23,834 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7a430b3e-23f6-4142-a481-16f83db80bfa, queue=awx_private_queue)
[2021-04-08 11:14:23,834: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7a430b3e-23f6-4142-a481-16f83db80bfa, queue=awx_private_queue)
2021-04-08 11:14:23,855 DEBUG awx.main.dispatch delivered 7a430b3e-23f6-4142-a481-16f83db80bfa to worker[178] qsize 0
2021-04-08 11:14:23,863 DEBUG awx.main.dispatch task 7a430b3e-23f6-4142-a481-16f83db80bfa starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:14:23,891 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:14:23,904 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:13:53.858538+00:00
2021-04-08 11:14:23,927 DEBUG awx.main.dispatch task 7a430b3e-23f6-4142-a481-16f83db80bfa is finished
2021-04-08 11:14:34,132 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(432c3a69-66e5-4598-b2c3-3eb10751f185, queue=awx_private_queue)
[2021-04-08 11:14:34,132: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(432c3a69-66e5-4598-b2c3-3eb10751f185, queue=awx_private_queue)
2021-04-08 11:14:34,145 DEBUG awx.main.dispatch delivered 432c3a69-66e5-4598-b2c3-3eb10751f185 to worker[178] qsize 0
2021-04-08 11:14:34,148 DEBUG awx.main.dispatch task 432c3a69-66e5-4598-b2c3-3eb10751f185 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:14:34,150 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:14:34,171 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:14:34,200 DEBUG awx.main.dispatch task 432c3a69-66e5-4598-b2c3-3eb10751f185 is finished
2021-04-08 11:14:53,503 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(446a534f-7d4c-41e4-81ff-2dacc59f8edc, queue=dev-awx-0)
[2021-04-08 11:14:53,503: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(446a534f-7d4c-41e4-81ff-2dacc59f8edc, queue=dev-awx-0)
2021-04-08 11:14:53,550 DEBUG awx.main.dispatch delivered 446a534f-7d4c-41e4-81ff-2dacc59f8edc to worker[177] qsize 0
2021-04-08 11:14:53,619 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(540f3212-a980-4e9b-9994-763a8513dc40, queue=dev-awx-0)
[2021-04-08 11:14:53,619: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(540f3212-a980-4e9b-9994-763a8513dc40, queue=dev-awx-0)
2021-04-08 11:14:53,634 DEBUG awx.main.dispatch delivered 540f3212-a980-4e9b-9994-763a8513dc40 to worker[179] qsize 0
2021-04-08 11:14:53,639 DEBUG awx.main.dispatch task 540f3212-a980-4e9b-9994-763a8513dc40 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:14:53,555 DEBUG awx.main.dispatch task 446a534f-7d4c-41e4-81ff-2dacc59f8edc starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:14:53,789 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:14:53,828 DEBUG awx.main.dispatch task 446a534f-7d4c-41e4-81ff-2dacc59f8edc is finished
2021-04-08 11:14:53,832 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4327bf13-19d6-4aa4-8324-b55ac96553fb, queue=awx_private_queue)
[2021-04-08 11:14:53,832: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4327bf13-19d6-4aa4-8324-b55ac96553fb, queue=awx_private_queue)
2021-04-08 11:14:53,680 DEBUG awx.main.dispatch task 540f3212-a980-4e9b-9994-763a8513dc40 is finished
2021-04-08 11:14:53,853 DEBUG awx.main.dispatch delivered 4327bf13-19d6-4aa4-8324-b55ac96553fb to worker[177] qsize 0
2021-04-08 11:14:53,857 DEBUG awx.main.dispatch task 4327bf13-19d6-4aa4-8324-b55ac96553fb starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:14:53,922 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:14:53,925 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:14:23.891945+00:00
2021-04-08 11:14:53,939 DEBUG awx.main.dispatch task 4327bf13-19d6-4aa4-8324-b55ac96553fb is finished
2021-04-08 11:14:54,132 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(30225d82-6ca0-499d-a5fa-06e09c9fcaca, queue=awx_private_queue)
[2021-04-08 11:14:54,132: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(30225d82-6ca0-499d-a5fa-06e09c9fcaca, queue=awx_private_queue)
2021-04-08 11:14:54,149 DEBUG awx.main.dispatch delivered 30225d82-6ca0-499d-a5fa-06e09c9fcaca to worker[176] qsize 0
2021-04-08 11:14:54,154 DEBUG awx.main.dispatch task 30225d82-6ca0-499d-a5fa-06e09c9fcaca starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:14:54,159 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:14:54,181 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:14:54,209 DEBUG awx.main.dispatch task 30225d82-6ca0-499d-a5fa-06e09c9fcaca is finished
RESULT 2
OKREADY
2021-04-08 11:15:14,142 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cda8a40e-502d-4346-9601-44e17d1a5682, queue=awx_private_queue)
[2021-04-08 11:15:14,142: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cda8a40e-502d-4346-9601-44e17d1a5682, queue=awx_private_queue)
2021-04-08 11:15:14,157 DEBUG awx.main.dispatch delivered cda8a40e-502d-4346-9601-44e17d1a5682 to worker[178] qsize 0
2021-04-08 11:15:14,161 DEBUG awx.main.dispatch task cda8a40e-502d-4346-9601-44e17d1a5682 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:15:14,165 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:15:23,832 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(751aa6fd-da19-4278-952c-145d4bb44019, queue=awx_private_queue)
[2021-04-08 11:15:23,832: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(751aa6fd-da19-4278-952c-145d4bb44019, queue=awx_private_queue)
2021-04-08 11:15:23,848 DEBUG awx.main.dispatch delivered 751aa6fd-da19-4278-952c-145d4bb44019 to worker[179] qsize 0
2021-04-08 11:15:23,850 DEBUG awx.main.dispatch task 751aa6fd-da19-4278-952c-145d4bb44019 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:15:34,142 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9bacb014-3313-4083-bcbe-68e20f5c88f9, queue=awx_private_queue)
[2021-04-08 11:15:34,142: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9bacb014-3313-4083-bcbe-68e20f5c88f9, queue=awx_private_queue)
2021-04-08 11:15:34,167 DEBUG awx.main.dispatch delivered 9bacb014-3313-4083-bcbe-68e20f5c88f9 to worker[176] qsize 0
2021-04-08 11:15:34,171 DEBUG awx.main.dispatch task 9bacb014-3313-4083-bcbe-68e20f5c88f9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:15:34,173 DEBUG awx.main.scheduler Running Tower task manager.
RESULT 2
OKREADY
RESULT 2
OKREADY
2021-04-08 11:17:13,296 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-08 11:17:13,295 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:15:53,510 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f7f3148f-38ce-4291-8f47-a08cd8ba414e, queue=dev-awx-0)
[2021-04-08 11:15:53,510: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f7f3148f-38ce-4291-8f47-a08cd8ba414e, queue=dev-awx-0)
2021-04-08 11:17:13,297 DEBUG awx.main.dispatch task cda8a40e-502d-4346-9601-44e17d1a5682 is finished
2021-04-08 11:17:13,527 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:17:13,563 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:14:53.923116+00:00
2021-04-08 11:17:13,407 DEBUG awx.main.dispatch delivered f7f3148f-38ce-4291-8f47-a08cd8ba414e to worker[177] qsize 0
2021-04-08 11:17:13,578 DEBUG awx.main.dispatch task f7f3148f-38ce-4291-8f47-a08cd8ba414e starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:17:13,583 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:17:13,364 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5edae099-bc89-439c-a8b4-aebcfc5df774, queue=dev-awx-0)
[2021-04-08 11:17:13,364: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5edae099-bc89-439c-a8b4-aebcfc5df774, queue=dev-awx-0)
2021-04-08 11:17:13,695 DEBUG awx.main.dispatch delivered 5edae099-bc89-439c-a8b4-aebcfc5df774 to worker[178] qsize 0
2021-04-08 11:17:13,695 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6aab82b2-36b3-4cec-b99f-8dee891c31a2, queue=awx_private_queue)
[2021-04-08 11:17:13,695: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6aab82b2-36b3-4cec-b99f-8dee891c31a2, queue=awx_private_queue)
2021-04-08 11:17:13,697 DEBUG awx.main.dispatch task 5edae099-bc89-439c-a8b4-aebcfc5df774 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:17:13,702 DEBUG awx.main.dispatch task 9bacb014-3313-4083-bcbe-68e20f5c88f9 is finished
2021-04-08 11:17:13,708 DEBUG awx.main.dispatch delivered 6aab82b2-36b3-4cec-b99f-8dee891c31a2 to worker[176] qsize 0
2021-04-08 11:17:13,710 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(748f60af-2578-41b7-8326-3e9819c5a992, queue=awx_private_queue)
[2021-04-08 11:17:13,710: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(748f60af-2578-41b7-8326-3e9819c5a992, queue=awx_private_queue)
2021-04-08 11:17:13,712 DEBUG awx.main.dispatch task 6aab82b2-36b3-4cec-b99f-8dee891c31a2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:17:13,732 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-04-08 11:17:13,732 DEBUG awx.main.dispatch task 6aab82b2-36b3-4cec-b99f-8dee891c31a2 is finished
2021-04-08 11:17:13,733 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(b47187c3-1b24-4b38-9161-d5ff8a339e8f, queue=dev-awx-0)
[2021-04-08 11:17:13,733: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(b47187c3-1b24-4b38-9161-d5ff8a339e8f, queue=dev-awx-0)
2021-04-08 11:17:13,743 WARNING awx.main.dispatch scaling up worker pid:192
2021-04-08 11:17:13,750 DEBUG awx.main.dispatch delivered 748f60af-2578-41b7-8326-3e9819c5a992 to worker[176] qsize 0
2021-04-08 11:17:13,753 DEBUG awx.main.dispatch task 748f60af-2578-41b7-8326-3e9819c5a992 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:17:13,754 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:17:13,753 DEBUG awx.main.dispatch delivered b47187c3-1b24-4b38-9161-d5ff8a339e8f to worker[192] qsize 0
2021-04-08 11:17:13,757 DEBUG awx.main.dispatch task b47187c3-1b24-4b38-9161-d5ff8a339e8f starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-04-08 11:17:13,760 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-04-08 11:17:13,797 DEBUG awx.main.dispatch task 5edae099-bc89-439c-a8b4-aebcfc5df774 is finished
2021-04-08 11:17:13,803 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:17:13,816 DEBUG awx.main.dispatch task b47187c3-1b24-4b38-9161-d5ff8a339e8f is finished
2021-04-08 11:17:13,855 WARNING awx.main.tasks Rejoining the cluster as instance dev-awx-0.
2021-04-08 11:17:13,855 DEBUG awx.main.dispatch task f7f3148f-38ce-4291-8f47-a08cd8ba414e is finished
2021-04-08 11:17:13,858 DEBUG awx.main.dispatch task 748f60af-2578-41b7-8326-3e9819c5a992 is finished
2021-04-08 11:17:13,990 DEBUG awx.main.dispatch task 751aa6fd-da19-4278-952c-145d4bb44019 is finished
2021-04-08 11:17:33,714 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(12e7679a-9a0c-454f-af9d-6e7dac32bec8, queue=awx_private_queue)
[2021-04-08 11:17:33,714: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(12e7679a-9a0c-454f-af9d-6e7dac32bec8, queue=awx_private_queue)
2021-04-08 11:17:33,730 DEBUG awx.main.dispatch delivered 12e7679a-9a0c-454f-af9d-6e7dac32bec8 to worker[177] qsize 0
2021-04-08 11:17:33,733 DEBUG awx.main.dispatch task 12e7679a-9a0c-454f-af9d-6e7dac32bec8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:17:33,734 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:17:33,750 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:17:33,786 DEBUG awx.main.dispatch task 12e7679a-9a0c-454f-af9d-6e7dac32bec8 is finished
2021-04-08 11:17:43,696 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d9fd3521-c8ee-4b0a-bb7c-af91ed5390d4, queue=awx_private_queue)
[2021-04-08 11:17:43,696: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d9fd3521-c8ee-4b0a-bb7c-af91ed5390d4, queue=awx_private_queue)
2021-04-08 11:17:43,716 DEBUG awx.main.dispatch delivered d9fd3521-c8ee-4b0a-bb7c-af91ed5390d4 to worker[176] qsize 0
2021-04-08 11:17:43,718 DEBUG awx.main.dispatch task d9fd3521-c8ee-4b0a-bb7c-af91ed5390d4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:17:43,726 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:17:43,729 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:17:13.529214+00:00
2021-04-08 11:17:43,741 DEBUG awx.main.dispatch task d9fd3521-c8ee-4b0a-bb7c-af91ed5390d4 is finished
2021-04-08 11:17:53,714 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a989232e-79ce-4804-a914-a59499ef0b89, queue=awx_private_queue)
[2021-04-08 11:17:53,714: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a989232e-79ce-4804-a914-a59499ef0b89, queue=awx_private_queue)
2021-04-08 11:17:53,731 DEBUG awx.main.dispatch delivered a989232e-79ce-4804-a914-a59499ef0b89 to worker[179] qsize 0
2021-04-08 11:17:53,733 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(3072e06f-d808-4cf2-9462-e996d861dd3a, queue=dev-awx-0)
2021-04-08 11:17:53,734 DEBUG awx.main.dispatch task a989232e-79ce-4804-a914-a59499ef0b89 starting awx.main.scheduler.tasks.run_task_manager(*[])
[2021-04-08 11:17:53,733: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(3072e06f-d808-4cf2-9462-e996d861dd3a, queue=dev-awx-0)
2021-04-08 11:17:53,735 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:17:53,753 WARNING awx.main.dispatch scaling down worker pid:176
2021-04-08 11:17:53,757 WARNING awx.main.dispatch worker exiting gracefully pid:176
2021-04-08 11:17:53,769 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:17:53,798 DEBUG awx.main.dispatch delivered 3072e06f-d808-4cf2-9462-e996d861dd3a to worker[178] qsize 0
2021-04-08 11:17:53,800 DEBUG awx.main.dispatch task 3072e06f-d808-4cf2-9462-e996d861dd3a starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:17:53,801 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:17:53,802 DEBUG awx.main.dispatch task a989232e-79ce-4804-a914-a59499ef0b89 is finished
2021-04-08 11:17:53,828 DEBUG awx.main.dispatch task 3072e06f-d808-4cf2-9462-e996d861dd3a is finished
RESULT 2
OKREADY
2021-04-08 11:18:13,370 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2960aacf-6d82-4dbd-b988-53939dad79e5, queue=dev-awx-0)
[2021-04-08 11:18:13,370: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2960aacf-6d82-4dbd-b988-53939dad79e5, queue=dev-awx-0)
2021-04-08 11:18:13,390 DEBUG awx.main.dispatch delivered 2960aacf-6d82-4dbd-b988-53939dad79e5 to worker[178] qsize 0
2021-04-08 11:18:13,394 DEBUG awx.main.dispatch task 2960aacf-6d82-4dbd-b988-53939dad79e5 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:18:13,411 DEBUG awx.main.dispatch task 2960aacf-6d82-4dbd-b988-53939dad79e5 is finished
2021-04-08 11:18:13,696 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4800bbb4-1e60-48cc-96f0-aac8a30a5c7d, queue=awx_private_queue)
[2021-04-08 11:18:13,696: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4800bbb4-1e60-48cc-96f0-aac8a30a5c7d, queue=awx_private_queue)
2021-04-08 11:18:13,714 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f1da349d-69f4-4b85-b874-cc03826f086a, queue=awx_private_queue)
[2021-04-08 11:18:13,714: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f1da349d-69f4-4b85-b874-cc03826f086a, queue=awx_private_queue)
2021-04-08 11:18:13,709 DEBUG awx.main.dispatch delivered 4800bbb4-1e60-48cc-96f0-aac8a30a5c7d to worker[177] qsize 0
2021-04-08 11:18:13,880 DEBUG awx.main.dispatch delivered f1da349d-69f4-4b85-b874-cc03826f086a to worker[192] qsize 0
2021-04-08 11:18:13,881 DEBUG awx.main.dispatch task 4800bbb4-1e60-48cc-96f0-aac8a30a5c7d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:18:13,887 DEBUG awx.main.dispatch task f1da349d-69f4-4b85-b874-cc03826f086a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:18:13,893 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:18:14,127 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:18:14,138 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:17:43.727109+00:00
2021-04-08 11:18:14,142 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:18:14,162 DEBUG awx.main.dispatch task 4800bbb4-1e60-48cc-96f0-aac8a30a5c7d is finished
2021-04-08 11:18:14,214 DEBUG awx.main.dispatch task f1da349d-69f4-4b85-b874-cc03826f086a is finished
2021-04-08 11:18:33,721 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a706e498-63a6-4ee1-b564-97421f5f15e8, queue=awx_private_queue)
[2021-04-08 11:18:33,721: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a706e498-63a6-4ee1-b564-97421f5f15e8, queue=awx_private_queue)
2021-04-08 11:18:33,739 DEBUG awx.main.dispatch delivered a706e498-63a6-4ee1-b564-97421f5f15e8 to worker[178] qsize 0
2021-04-08 11:18:33,742 DEBUG awx.main.dispatch task a706e498-63a6-4ee1-b564-97421f5f15e8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:18:33,743 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:18:43,696 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b5730335-e13d-46cc-a2ef-9288302d0146, queue=awx_private_queue)
[2021-04-08 11:18:43,696: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b5730335-e13d-46cc-a2ef-9288302d0146, queue=awx_private_queue)
2021-04-08 11:18:43,715 DEBUG awx.main.dispatch delivered b5730335-e13d-46cc-a2ef-9288302d0146 to worker[192] qsize 0
2021-04-08 11:18:43,719 DEBUG awx.main.dispatch task b5730335-e13d-46cc-a2ef-9288302d0146 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:18:53,719 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ea3b78bf-e0ce-40b3-b359-d89265d130f8, queue=awx_private_queue)
[2021-04-08 11:18:53,719: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ea3b78bf-e0ce-40b3-b359-d89265d130f8, queue=awx_private_queue)
2021-04-08 11:18:53,734 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(de5c5c80-8fdf-4ec6-a2b6-ab6e4098f515, queue=dev-awx-0)
[2021-04-08 11:18:53,734: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(de5c5c80-8fdf-4ec6-a2b6-ab6e4098f515, queue=dev-awx-0)
2021-04-08 11:18:53,732 DEBUG awx.main.dispatch delivered ea3b78bf-e0ce-40b3-b359-d89265d130f8 to worker[177] qsize 0
2021-04-08 11:18:53,736 DEBUG awx.main.dispatch task ea3b78bf-e0ce-40b3-b359-d89265d130f8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:18:53,738 DEBUG awx.main.scheduler Running Tower task manager.
RESULT 2
OKREADY
2021-04-08 11:19:13,373 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d0371b0f-18e3-4d0b-9cd8-79c777ad971c, queue=dev-awx-0)
[2021-04-08 11:19:13,373: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d0371b0f-18e3-4d0b-9cd8-79c777ad971c, queue=dev-awx-0)
RESULT 2
OKREADY
2021-04-08 11:20:09,425 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:20:09,451 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:19:13,696 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(88ac1b79-0306-4d4c-96b5-bf381a4130a8, queue=awx_private_queue)
[2021-04-08 11:19:13,696: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(88ac1b79-0306-4d4c-96b5-bf381a4130a8, queue=awx_private_queue)
2021-04-08 11:20:09,500 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:18:14.128356+00:00
2021-04-08 11:20:09,531 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2ecad7d7-cdb2-467a-8fb3-702fe1e1ed4b, queue=awx_private_queue)
[2021-04-08 11:20:09,531: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2ecad7d7-cdb2-467a-8fb3-702fe1e1ed4b, queue=awx_private_queue)
2021-04-08 11:20:09,554 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(7d80cbcf-4ec7-4950-8658-04fb8ce554fa, queue=dev-awx-0)
[2021-04-08 11:20:09,554: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(7d80cbcf-4ec7-4950-8658-04fb8ce554fa, queue=dev-awx-0)
2021-04-08 11:20:09,489 DEBUG awx.main.dispatch task ea3b78bf-e0ce-40b3-b359-d89265d130f8 is finished
2021-04-08 11:20:09,705 DEBUG awx.main.dispatch delivered de5c5c80-8fdf-4ec6-a2b6-ab6e4098f515 to worker[177] qsize 0
2021-04-08 11:20:09,712 DEBUG awx.main.dispatch task de5c5c80-8fdf-4ec6-a2b6-ab6e4098f515 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:20:09,713 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:20:09,713 DEBUG awx.main.dispatch delivered d0371b0f-18e3-4d0b-9cd8-79c777ad971c to worker[179] qsize 0
2021-04-08 11:20:09,717 DEBUG awx.main.dispatch task d0371b0f-18e3-4d0b-9cd8-79c777ad971c starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:20:09,735 WARNING awx.main.dispatch scaling up worker pid:195
2021-04-08 11:20:09,739 DEBUG awx.main.dispatch delivered 88ac1b79-0306-4d4c-96b5-bf381a4130a8 to worker[195] qsize 0
2021-04-08 11:20:09,742 DEBUG awx.main.dispatch task 88ac1b79-0306-4d4c-96b5-bf381a4130a8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:20:09,748 WARNING awx.main.dispatch scaling up worker pid:197
2021-04-08 11:20:09,752 DEBUG awx.main.dispatch delivered 2ecad7d7-cdb2-467a-8fb3-702fe1e1ed4b to worker[197] qsize 0
2021-04-08 11:20:09,763 DEBUG awx.main.dispatch task 2ecad7d7-cdb2-467a-8fb3-702fe1e1ed4b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:20:09,766 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:20:09,872 DEBUG awx.main.dispatch task b5730335-e13d-46cc-a2ef-9288302d0146 is finished
2021-04-08 11:20:09,919 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:20:09,945 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:20:09.425945+00:00
2021-04-08 11:20:09,999 DEBUG awx.main.dispatch task d0371b0f-18e3-4d0b-9cd8-79c777ad971c is finished
2021-04-08 11:20:10,002 DEBUG awx.main.dispatch task 88ac1b79-0306-4d4c-96b5-bf381a4130a8 is finished
2021-04-08 11:20:10,035 DEBUG awx.main.dispatch delivered 7d80cbcf-4ec7-4950-8658-04fb8ce554fa to worker[192] qsize 0
2021-04-08 11:20:10,039 DEBUG awx.main.dispatch task 7d80cbcf-4ec7-4950-8658-04fb8ce554fa starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:20:10,040 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:20:10,097 WARNING awx.main.tasks Rejoining the cluster as instance dev-awx-0.
2021-04-08 11:20:10,097 DEBUG awx.main.dispatch task de5c5c80-8fdf-4ec6-a2b6-ab6e4098f515 is finished
2021-04-08 11:20:10,137 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:20:10,201 DEBUG awx.main.dispatch task 7d80cbcf-4ec7-4950-8658-04fb8ce554fa is finished
2021-04-08 11:20:10,251 DEBUG awx.main.dispatch task 2ecad7d7-cdb2-467a-8fb3-702fe1e1ed4b is finished
2021-04-08 11:20:10,721 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:20:10,915 DEBUG awx.main.dispatch task a706e498-63a6-4ee1-b564-97421f5f15e8 is finished
2021-04-08 11:20:13,373 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8c26cad7-ac35-4337-a67f-14e02535dde7, queue=dev-awx-0)
[2021-04-08 11:20:13,373: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8c26cad7-ac35-4337-a67f-14e02535dde7, queue=dev-awx-0)
2021-04-08 11:20:13,389 DEBUG awx.main.dispatch delivered 8c26cad7-ac35-4337-a67f-14e02535dde7 to worker[177] qsize 0
2021-04-08 11:20:13,391 DEBUG awx.main.dispatch task 8c26cad7-ac35-4337-a67f-14e02535dde7 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:20:13,413 DEBUG awx.main.dispatch task 8c26cad7-ac35-4337-a67f-14e02535dde7 is finished
2021-04-08 11:20:29,529 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9a0f577e-8a64-4504-8450-75280e66c320, queue=awx_private_queue)
[2021-04-08 11:20:29,529: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9a0f577e-8a64-4504-8450-75280e66c320, queue=awx_private_queue)
2021-04-08 11:20:29,543 DEBUG awx.main.dispatch delivered 9a0f577e-8a64-4504-8450-75280e66c320 to worker[178] qsize 0
2021-04-08 11:20:29,546 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(50240a14-5f66-4cc4-9363-7527a08674a2, queue=awx_private_queue)
[2021-04-08 11:20:29,546: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(50240a14-5f66-4cc4-9363-7527a08674a2, queue=awx_private_queue)
2021-04-08 11:20:29,547 DEBUG awx.main.dispatch task 9a0f577e-8a64-4504-8450-75280e66c320 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:20:29,549 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:20:29,566 DEBUG awx.main.dispatch delivered 50240a14-5f66-4cc4-9363-7527a08674a2 to worker[177] qsize 0
2021-04-08 11:20:29,569 DEBUG awx.main.dispatch task 50240a14-5f66-4cc4-9363-7527a08674a2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:20:29,571 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:20:29,577 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:20:29,579 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:20:09.920293+00:00
2021-04-08 11:20:29,593 DEBUG awx.main.dispatch task 50240a14-5f66-4cc4-9363-7527a08674a2 is finished
2021-04-08 11:20:29,598 DEBUG awx.main.dispatch task 9a0f577e-8a64-4504-8450-75280e66c320 is finished
2021-04-08 11:20:49,547 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8fb37ac0-756a-4993-a6b0-6bc4dea8d300, queue=awx_private_queue)
[2021-04-08 11:20:49,547: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8fb37ac0-756a-4993-a6b0-6bc4dea8d300, queue=awx_private_queue)
2021-04-08 11:20:49,569 DEBUG awx.main.dispatch delivered 8fb37ac0-756a-4993-a6b0-6bc4dea8d300 to worker[179] qsize 0
2021-04-08 11:20:49,574 DEBUG awx.main.dispatch task 8fb37ac0-756a-4993-a6b0-6bc4dea8d300 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:20:49,579 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:20:49,603 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:20:49,635 DEBUG awx.main.dispatch task 8fb37ac0-756a-4993-a6b0-6bc4dea8d300 is finished
2021-04-08 11:20:59,547 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e5973a70-d88b-410d-a3ea-df417a6e49a8, queue=awx_private_queue)
[2021-04-08 11:20:59,547: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e5973a70-d88b-410d-a3ea-df417a6e49a8, queue=awx_private_queue)
2021-04-08 11:20:59,565 DEBUG awx.main.dispatch delivered e5973a70-d88b-410d-a3ea-df417a6e49a8 to worker[197] qsize 0
2021-04-08 11:20:59,570 DEBUG awx.main.dispatch task e5973a70-d88b-410d-a3ea-df417a6e49a8 starting awx.main.tasks.awx_periodic_scheduler(*[])
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
2021-04-08 11:26:52,112 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:26:52,120 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:20:29.577399+00:00
2021-04-08 11:21:09,555 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(50e24d37-7b8a-4dc2-9f3a-0c4520c2bbc7, queue=dev-awx-0)
[2021-04-08 11:21:09,555: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(50e24d37-7b8a-4dc2-9f3a-0c4520c2bbc7, queue=dev-awx-0)
2021-04-08 11:26:52,143 WARNING awx.main.dispatch scaling down worker pid:177
2021-04-08 11:26:52,330 WARNING awx.main.dispatch scaling down worker pid:178
2021-04-08 11:26:52,331 WARNING awx.main.dispatch worker exiting gracefully pid:177
2021-04-08 11:26:52,334 WARNING awx.main.dispatch worker exiting gracefully pid:178
2021-04-08 11:26:52,353 DEBUG awx.main.dispatch delivered 50e24d37-7b8a-4dc2-9f3a-0c4520c2bbc7 to worker[195] qsize 0
2021-04-08 11:26:52,358 DEBUG awx.main.dispatch task 50e24d37-7b8a-4dc2-9f3a-0c4520c2bbc7 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-08 11:26:52,364 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-08 11:26:52,150 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5a829cad-9376-4922-8e5a-2e3304f618cf, queue=awx_private_queue)
[2021-04-08 11:26:52,150: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5a829cad-9376-4922-8e5a-2e3304f618cf, queue=awx_private_queue)
2021-04-08 11:26:52,426 DEBUG awx.main.dispatch delivered 5a829cad-9376-4922-8e5a-2e3304f618cf to worker[179] qsize 0
2021-04-08 11:26:52,427 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(78186095-2d33-42ea-b23f-d62a656718e1, queue=dev-awx-0)
[2021-04-08 11:26:52,427: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(78186095-2d33-42ea-b23f-d62a656718e1, queue=dev-awx-0)
2021-04-08 11:26:52,431 DEBUG awx.main.dispatch task 5a829cad-9376-4922-8e5a-2e3304f618cf starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:26:52,437 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:26:52,258 DEBUG awx.main.dispatch task e5973a70-d88b-410d-a3ea-df417a6e49a8 is finished
2021-04-08 11:26:52,444 DEBUG awx.main.dispatch delivered 78186095-2d33-42ea-b23f-d62a656718e1 to worker[192] qsize 0
2021-04-08 11:26:52,444 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bff53c73-7206-43dd-bce6-726a13aaedee, queue=awx_private_queue)
[2021-04-08 11:26:52,444: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bff53c73-7206-43dd-bce6-726a13aaedee, queue=awx_private_queue)
2021-04-08 11:26:52,448 DEBUG awx.main.dispatch task 78186095-2d33-42ea-b23f-d62a656718e1 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:26:52,468 DEBUG awx.main.dispatch delivered bff53c73-7206-43dd-bce6-726a13aaedee to worker[197] qsize 0
2021-04-08 11:26:52,470 DEBUG awx.main.dispatch task bff53c73-7206-43dd-bce6-726a13aaedee starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:26:52,596 WARNING awx.main.tasks Rejoining the cluster as instance dev-awx-0.
2021-04-08 11:26:52,597 DEBUG awx.main.dispatch task 50e24d37-7b8a-4dc2-9f3a-0c4520c2bbc7 is finished
2021-04-08 11:26:52,620 DEBUG awx.main.scheduler Starting Scheduler
2021-04-08 11:26:52,715 DEBUG awx.main.dispatch task 78186095-2d33-42ea-b23f-d62a656718e1 is finished
2021-04-08 11:26:52,763 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:26:52,765 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 11:26:52.113219+00:00
2021-04-08 11:26:52,784 DEBUG awx.main.dispatch task 5a829cad-9376-4922-8e5a-2e3304f618cf is finished
2021-04-08 11:26:52,786 DEBUG awx.main.dispatch task bff53c73-7206-43dd-bce6-726a13aaedee is finished
RESULT 2
OKREADY
2021-04-08 11:27:12,161 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c5d5d5e3-1eb9-4dfa-8256-11a9c8f241f5, queue=awx_private_queue)
[2021-04-08 11:27:12,161: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c5d5d5e3-1eb9-4dfa-8256-11a9c8f241f5, queue=awx_private_queue)
2021-04-08 11:27:12,175 DEBUG awx.main.dispatch delivered c5d5d5e3-1eb9-4dfa-8256-11a9c8f241f5 to worker[192] qsize 0
2021-04-08 11:27:12,177 DEBUG awx.main.dispatch task c5d5d5e3-1eb9-4dfa-8256-11a9c8f241f5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:27:12,179 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:27:12,183 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL: the database system is shutting down
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: FATAL: the database system is shutting down
2021-04-08 11:27:12,191 DEBUG awx.main.dispatch task c5d5d5e3-1eb9-4dfa-8256-11a9c8f241f5 is finished
2021-04-08 11:27:13,733 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(94314c94-cc4d-4a91-a908-f8d3fa46fe4f, queue=dev-awx-0)
[2021-04-08 11:27:13,733: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(94314c94-cc4d-4a91-a908-f8d3fa46fe4f, queue=dev-awx-0)
2021-04-08 11:27:13,747 DEBUG awx.main.dispatch delivered 94314c94-cc4d-4a91-a908-f8d3fa46fe4f to worker[195] qsize 0
2021-04-08 11:27:13,750 DEBUG awx.main.dispatch task 94314c94-cc4d-4a91-a908-f8d3fa46fe4f starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-04-08 11:27:13,755 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-04-08 11:27:13,769 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL: the database system is shutting down
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 469, in awx_isolated_heartbeat
with transaction.atomic():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/transaction.py", line 175, in __enter__
if not connection.get_autocommit():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 379, in get_autocommit
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: FATAL: the database system is shutting down
2021-04-08 11:27:13,776 DEBUG awx.main.dispatch task 94314c94-cc4d-4a91-a908-f8d3fa46fe4f is finished
2021-04-08 11:27:22,444 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d6a28705-e11a-4135-a5e7-dc5f18bc8483, queue=awx_private_queue)
[2021-04-08 11:27:22,444: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d6a28705-e11a-4135-a5e7-dc5f18bc8483, queue=awx_private_queue)
2021-04-08 11:27:22,459 DEBUG awx.main.dispatch delivered d6a28705-e11a-4135-a5e7-dc5f18bc8483 to worker[197] qsize 0
2021-04-08 11:27:22,462 DEBUG awx.main.dispatch task d6a28705-e11a-4135-a5e7-dc5f18bc8483 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:27:22,468 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL: the database system is shutting down
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: FATAL: the database system is shutting down
2021-04-08 11:27:22,475 DEBUG awx.main.dispatch task d6a28705-e11a-4135-a5e7-dc5f18bc8483 is finished
2021-04-08 11:27:32,160 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(250f79e6-a019-4e4b-89cb-d3e06490b964, queue=awx_private_queue)
[2021-04-08 11:27:32,160: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(250f79e6-a019-4e4b-89cb-d3e06490b964, queue=awx_private_queue)
2021-04-08 11:27:32,174 DEBUG awx.main.dispatch delivered 250f79e6-a019-4e4b-89cb-d3e06490b964 to worker[195] qsize 0
2021-04-08 11:27:32,175 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(fcc3a636-0979-499c-b5f8-0fdf938b3836, queue=dev-awx-0)
[2021-04-08 11:27:32,175: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(fcc3a636-0979-499c-b5f8-0fdf938b3836, queue=dev-awx-0)
2021-04-08 11:27:32,178 DEBUG awx.main.dispatch task 250f79e6-a019-4e4b-89cb-d3e06490b964 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:27:32,180 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:27:32,183 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.152.198) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.152.198) and accepting
TCP/IP connections on port 5432?
2021-04-08 11:27:32,187 DEBUG awx.main.dispatch task 250f79e6-a019-4e4b-89cb-d3e06490b964 is finished
2021-04-08 11:27:32,197 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 82, in _execute
return self.cursor.execute(sql)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 123, in register
with advisory_lock('instance_registration_%s' % hostname):
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 66, in advisory_lock
cursor.execute(command)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 82, in _execute
return self.cursor.execute(sql)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2021-04-08 11:27:52,163 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2f269856-db36-41fa-88b0-c0f8b1855c8a, queue=awx_private_queue)
[2021-04-08 11:27:52,163: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2f269856-db36-41fa-88b0-c0f8b1855c8a, queue=awx_private_queue)
2021-04-08 11:27:52,204 DEBUG awx.main.dispatch delivered 2f269856-db36-41fa-88b0-c0f8b1855c8a to worker[179] qsize 0
2021-04-08 11:27:52,231 DEBUG awx.main.dispatch task 2f269856-db36-41fa-88b0-c0f8b1855c8a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:27:52,251 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:27:52,253 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:27:52,257 DEBUG awx.main.dispatch task 2f269856-db36-41fa-88b0-c0f8b1855c8a is finished
2021-04-08 11:27:52,428 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(cb7b1e84-e9e9-47dc-b53f-a460ce7b9b0d, queue=dev-awx-0)
[2021-04-08 11:27:52,428: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(cb7b1e84-e9e9-47dc-b53f-a460ce7b9b0d, queue=dev-awx-0)
2021-04-08 11:27:52,451 DEBUG awx.main.dispatch delivered cb7b1e84-e9e9-47dc-b53f-a460ce7b9b0d to worker[192] qsize 0
2021-04-08 11:27:52,454 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e1594809-0ee2-4d47-9e38-44ca67a5a27a, queue=awx_private_queue)
[2021-04-08 11:27:52,454: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e1594809-0ee2-4d47-9e38-44ca67a5a27a, queue=awx_private_queue)
2021-04-08 11:27:52,458 DEBUG awx.main.dispatch task cb7b1e84-e9e9-47dc-b53f-a460ce7b9b0d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:27:52,475 DEBUG awx.main.dispatch delivered e1594809-0ee2-4d47-9e38-44ca67a5a27a to worker[195] qsize 0
2021-04-08 11:27:52,484 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139801040410176_2" does not exist
2021-04-08 11:27:52,490 DEBUG awx.main.dispatch task cb7b1e84-e9e9-47dc-b53f-a460ce7b9b0d is finished
2021-04-08 11:27:52,479 DEBUG awx.main.dispatch task e1594809-0ee2-4d47-9e38-44ca67a5a27a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:27:52,500 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:27:52,502 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:27:52,507 DEBUG awx.main.dispatch task e1594809-0ee2-4d47-9e38-44ca67a5a27a is finished
RESULT 2
OKREADY
2021-04-08 11:28:12,172 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9c130cad-fc0b-4313-9bb0-f60d95a30c84, queue=awx_private_queue)
[2021-04-08 11:28:12,172: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9c130cad-fc0b-4313-9bb0-f60d95a30c84, queue=awx_private_queue)
2021-04-08 11:28:12,200 DEBUG awx.main.dispatch delivered 9c130cad-fc0b-4313-9bb0-f60d95a30c84 to worker[192] qsize 0
2021-04-08 11:28:12,211 DEBUG awx.main.dispatch task 9c130cad-fc0b-4313-9bb0-f60d95a30c84 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:28:12,223 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:28:12,225 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:28:12,230 DEBUG awx.main.dispatch task 9c130cad-fc0b-4313-9bb0-f60d95a30c84 is finished
2021-04-08 11:28:22,456 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5a3166b8-8f0a-4a87-a1c2-8f7a89bf9fc5, queue=awx_private_queue)
[2021-04-08 11:28:22,456: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5a3166b8-8f0a-4a87-a1c2-8f7a89bf9fc5, queue=awx_private_queue)
2021-04-08 11:28:22,494 DEBUG awx.main.dispatch delivered 5a3166b8-8f0a-4a87-a1c2-8f7a89bf9fc5 to worker[179] qsize 0
2021-04-08 11:28:22,505 DEBUG awx.main.dispatch task 5a3166b8-8f0a-4a87-a1c2-8f7a89bf9fc5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:28:22,523 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:28:22,529 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:28:22,537 DEBUG awx.main.dispatch task 5a3166b8-8f0a-4a87-a1c2-8f7a89bf9fc5 is finished
2021-04-08 11:28:32,176 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e2c68d4c-c2c9-4fbc-95fd-454c3b268df3, queue=dev-awx-0)
[2021-04-08 11:28:32,176: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e2c68d4c-c2c9-4fbc-95fd-454c3b268df3, queue=dev-awx-0)
2021-04-08 11:28:32,200 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1e5dc4ac-2a48-4dd7-8880-c6e6ce231fc6, queue=awx_private_queue)
[2021-04-08 11:28:32,200: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1e5dc4ac-2a48-4dd7-8880-c6e6ce231fc6, queue=awx_private_queue)
2021-04-08 11:28:32,213 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-08 11:28:32,226 DEBUG awx.main.dispatch delivered 1e5dc4ac-2a48-4dd7-8880-c6e6ce231fc6 to worker[197] qsize 0
2021-04-08 11:28:32,232 DEBUG awx.main.dispatch task 1e5dc4ac-2a48-4dd7-8880-c6e6ce231fc6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:28:32,258 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:28:32,259 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:28:32,263 DEBUG awx.main.dispatch task 1e5dc4ac-2a48-4dd7-8880-c6e6ce231fc6 is finished
2021-04-08 11:28:52,217 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d138df24-684a-4242-b921-c435952a6da7, queue=awx_private_queue)
[2021-04-08 11:28:52,217: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d138df24-684a-4242-b921-c435952a6da7, queue=awx_private_queue)
2021-04-08 11:28:52,265 DEBUG awx.main.dispatch delivered d138df24-684a-4242-b921-c435952a6da7 to worker[179] qsize 0
2021-04-08 11:28:52,295 DEBUG awx.main.dispatch task d138df24-684a-4242-b921-c435952a6da7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:28:52,322 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:28:52,327 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:28:52,333 DEBUG awx.main.dispatch task d138df24-684a-4242-b921-c435952a6da7 is finished
2021-04-08 11:28:52,429 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f4fb43d1-b410-4b97-aacd-778e12951a56, queue=dev-awx-0)
[2021-04-08 11:28:52,429: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f4fb43d1-b410-4b97-aacd-778e12951a56, queue=dev-awx-0)
2021-04-08 11:28:52,486 DEBUG awx.main.dispatch delivered f4fb43d1-b410-4b97-aacd-778e12951a56 to worker[192] qsize 0
2021-04-08 11:28:52,489 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0e0744da-9ff9-4708-b924-96e7ebcf207e, queue=awx_private_queue)
[2021-04-08 11:28:52,489: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0e0744da-9ff9-4708-b924-96e7ebcf207e, queue=awx_private_queue)
2021-04-08 11:28:52,506 DEBUG awx.main.dispatch task f4fb43d1-b410-4b97-aacd-778e12951a56 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:28:52,529 DEBUG awx.main.dispatch delivered 0e0744da-9ff9-4708-b924-96e7ebcf207e to worker[197] qsize 0
2021-04-08 11:28:52,531 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139801040410176_3" does not exist
2021-04-08 11:28:52,548 DEBUG awx.main.dispatch task f4fb43d1-b410-4b97-aacd-778e12951a56 is finished
2021-04-08 11:28:52,538 DEBUG awx.main.dispatch task 0e0744da-9ff9-4708-b924-96e7ebcf207e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:28:52,578 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:28:52,584 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:28:52,589 DEBUG awx.main.dispatch task 0e0744da-9ff9-4708-b924-96e7ebcf207e is finished
RESULT 2
OKREADY
2021-04-08 11:29:12,216 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e0eed08b-bcdc-456e-b5ac-f5c68b93167d, queue=awx_private_queue)
[2021-04-08 11:29:12,216: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e0eed08b-bcdc-456e-b5ac-f5c68b93167d, queue=awx_private_queue)
2021-04-08 11:29:12,263 DEBUG awx.main.dispatch delivered e0eed08b-bcdc-456e-b5ac-f5c68b93167d to worker[195] qsize 0
2021-04-08 11:29:12,274 DEBUG awx.main.dispatch task e0eed08b-bcdc-456e-b5ac-f5c68b93167d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:29:12,294 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:29:12,296 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:29:12,301 DEBUG awx.main.dispatch task e0eed08b-bcdc-456e-b5ac-f5c68b93167d is finished
2021-04-08 11:29:22,488 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6f02478f-2749-405f-b062-868814a4092e, queue=awx_private_queue)
[2021-04-08 11:29:22,488: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6f02478f-2749-405f-b062-868814a4092e, queue=awx_private_queue)
2021-04-08 11:29:22,509 DEBUG awx.main.dispatch delivered 6f02478f-2749-405f-b062-868814a4092e to worker[195] qsize 0
2021-04-08 11:29:22,515 DEBUG awx.main.dispatch task 6f02478f-2749-405f-b062-868814a4092e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:29:22,526 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:29:22,528 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:29:22,532 DEBUG awx.main.dispatch task 6f02478f-2749-405f-b062-868814a4092e is finished
2021-04-08 11:29:32,177 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(90446c21-0611-4cfd-8760-0103f7eb7733, queue=dev-awx-0)
[2021-04-08 11:29:32,177: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(90446c21-0611-4cfd-8760-0103f7eb7733, queue=dev-awx-0)
2021-04-08 11:29:32,213 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1a529178-8f56-411a-a783-35886a76081e, queue=awx_private_queue)
[2021-04-08 11:29:32,213: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1a529178-8f56-411a-a783-35886a76081e, queue=awx_private_queue)
2021-04-08 11:29:32,213 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-08 11:29:32,239 DEBUG awx.main.dispatch delivered 1a529178-8f56-411a-a783-35886a76081e to worker[195] qsize 0
2021-04-08 11:29:32,244 DEBUG awx.main.dispatch task 1a529178-8f56-411a-a783-35886a76081e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:29:32,254 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:29:32,255 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:29:32,258 DEBUG awx.main.dispatch task 1a529178-8f56-411a-a783-35886a76081e is finished
2021-04-08 11:29:52,219 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5b0e6b1a-ac65-408f-83ea-9733f3807718, queue=awx_private_queue)
[2021-04-08 11:29:52,219: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5b0e6b1a-ac65-408f-83ea-9733f3807718, queue=awx_private_queue)
2021-04-08 11:29:52,252 DEBUG awx.main.dispatch delivered 5b0e6b1a-ac65-408f-83ea-9733f3807718 to worker[192] qsize 0
2021-04-08 11:29:52,263 DEBUG awx.main.dispatch task 5b0e6b1a-ac65-408f-83ea-9733f3807718 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:29:52,290 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:29:52,292 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:29:52,297 DEBUG awx.main.dispatch task 5b0e6b1a-ac65-408f-83ea-9733f3807718 is finished
2021-04-08 11:29:52,429 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(6aac3abf-4898-439c-8dd5-d1bcb9e4f717, queue=dev-awx-0)
[2021-04-08 11:29:52,429: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(6aac3abf-4898-439c-8dd5-d1bcb9e4f717, queue=dev-awx-0)
2021-04-08 11:29:52,483 DEBUG awx.main.dispatch delivered 6aac3abf-4898-439c-8dd5-d1bcb9e4f717 to worker[179] qsize 0
2021-04-08 11:29:52,490 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4e75caad-fd41-4c0a-b095-cce975518dc5, queue=awx_private_queue)
[2021-04-08 11:29:52,490: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4e75caad-fd41-4c0a-b095-cce975518dc5, queue=awx_private_queue)
2021-04-08 11:29:52,506 DEBUG awx.main.dispatch task 6aac3abf-4898-439c-8dd5-d1bcb9e4f717 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:29:52,536 DEBUG awx.main.dispatch delivered 4e75caad-fd41-4c0a-b095-cce975518dc5 to worker[192] qsize 0
2021-04-08 11:29:52,539 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139801040410176_9" does not exist
2021-04-08 11:29:52,546 DEBUG awx.main.dispatch task 6aac3abf-4898-439c-8dd5-d1bcb9e4f717 is finished
2021-04-08 11:29:52,547 DEBUG awx.main.dispatch task 4e75caad-fd41-4c0a-b095-cce975518dc5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:29:52,591 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:29:52,598 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:29:52,607 DEBUG awx.main.dispatch task 4e75caad-fd41-4c0a-b095-cce975518dc5 is finished
RESULT 2
OKREADY
2021-04-08 11:30:12,234 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e6e4e875-8b66-421c-886c-659415eec1c0, queue=awx_private_queue)
[2021-04-08 11:30:12,234: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e6e4e875-8b66-421c-886c-659415eec1c0, queue=awx_private_queue)
2021-04-08 11:30:12,267 DEBUG awx.main.dispatch delivered e6e4e875-8b66-421c-886c-659415eec1c0 to worker[192] qsize 0
2021-04-08 11:30:12,277 DEBUG awx.main.dispatch task e6e4e875-8b66-421c-886c-659415eec1c0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:30:12,288 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:30:12,289 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:30:12,293 DEBUG awx.main.dispatch task e6e4e875-8b66-421c-886c-659415eec1c0 is finished
2021-04-08 11:30:22,490 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c667bb56-7634-49a9-b0af-966a27ba6779, queue=awx_private_queue)
[2021-04-08 11:30:22,490: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c667bb56-7634-49a9-b0af-966a27ba6779, queue=awx_private_queue)
2021-04-08 11:30:22,523 DEBUG awx.main.dispatch delivered c667bb56-7634-49a9-b0af-966a27ba6779 to worker[197] qsize 0
2021-04-08 11:30:22,536 DEBUG awx.main.dispatch task c667bb56-7634-49a9-b0af-966a27ba6779 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:30:22,560 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:30:22,562 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:30:22,567 DEBUG awx.main.dispatch task c667bb56-7634-49a9-b0af-966a27ba6779 is finished
2021-04-08 11:30:32,177 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b94b06f2-7c1c-46ab-ba97-a5afd875889e, queue=dev-awx-0)
[2021-04-08 11:30:32,177: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b94b06f2-7c1c-46ab-ba97-a5afd875889e, queue=dev-awx-0)
2021-04-08 11:30:32,225 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-08 11:30:32,230 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ca7feb1f-9dcd-448f-9a2b-6dc44c651642, queue=awx_private_queue)
[2021-04-08 11:30:32,230: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ca7feb1f-9dcd-448f-9a2b-6dc44c651642, queue=awx_private_queue)
2021-04-08 11:30:32,257 DEBUG awx.main.dispatch delivered ca7feb1f-9dcd-448f-9a2b-6dc44c651642 to worker[195] qsize 0
2021-04-08 11:30:32,266 DEBUG awx.main.dispatch task ca7feb1f-9dcd-448f-9a2b-6dc44c651642 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:30:32,290 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:30:32,292 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:30:32,297 DEBUG awx.main.dispatch task ca7feb1f-9dcd-448f-9a2b-6dc44c651642 is finished
2021-04-08 11:30:52,238 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(73f442a4-dfe1-4898-b224-20dcb04ca579, queue=awx_private_queue)
[2021-04-08 11:30:52,238: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(73f442a4-dfe1-4898-b224-20dcb04ca579, queue=awx_private_queue)
2021-04-08 11:30:52,265 DEBUG awx.main.dispatch delivered 73f442a4-dfe1-4898-b224-20dcb04ca579 to worker[197] qsize 0
2021-04-08 11:30:52,306 DEBUG awx.main.dispatch task 73f442a4-dfe1-4898-b224-20dcb04ca579 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:30:52,320 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:30:52,323 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:30:52,335 DEBUG awx.main.dispatch task 73f442a4-dfe1-4898-b224-20dcb04ca579 is finished
2021-04-08 11:30:52,446 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(54db91bd-fe09-4541-87ec-6f5566ab6eef, queue=dev-awx-0)
[2021-04-08 11:30:52,446: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(54db91bd-fe09-4541-87ec-6f5566ab6eef, queue=dev-awx-0)
2021-04-08 11:30:52,471 DEBUG awx.main.dispatch delivered 54db91bd-fe09-4541-87ec-6f5566ab6eef to worker[192] qsize 0
2021-04-08 11:30:52,479 DEBUG awx.main.dispatch task 54db91bd-fe09-4541-87ec-6f5566ab6eef starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:30:52,502 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139801040410176_4" does not exist
2021-04-08 11:30:52,507 DEBUG awx.main.dispatch task 54db91bd-fe09-4541-87ec-6f5566ab6eef is finished
2021-04-08 11:30:52,491 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(eb1ca88c-b8a3-4fe8-ad64-5ca1950950d4, queue=awx_private_queue)
[2021-04-08 11:30:52,491: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(eb1ca88c-b8a3-4fe8-ad64-5ca1950950d4, queue=awx_private_queue)
2021-04-08 11:30:52,524 DEBUG awx.main.dispatch delivered eb1ca88c-b8a3-4fe8-ad64-5ca1950950d4 to worker[197] qsize 0
2021-04-08 11:30:52,559 DEBUG awx.main.dispatch task eb1ca88c-b8a3-4fe8-ad64-5ca1950950d4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:30:52,613 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:30:52,616 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:30:52,620 DEBUG awx.main.dispatch task eb1ca88c-b8a3-4fe8-ad64-5ca1950950d4 is finished
RESULT 2
OKREADY
2021-04-08 11:31:12,258 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(772f5571-dda8-45d9-a616-781f807370cf, queue=awx_private_queue)
[2021-04-08 11:31:12,258: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(772f5571-dda8-45d9-a616-781f807370cf, queue=awx_private_queue)
2021-04-08 11:31:12,292 DEBUG awx.main.dispatch delivered 772f5571-dda8-45d9-a616-781f807370cf to worker[197] qsize 0
2021-04-08 11:31:12,310 DEBUG awx.main.dispatch task 772f5571-dda8-45d9-a616-781f807370cf starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:31:12,331 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:31:12,333 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:31:12,337 DEBUG awx.main.dispatch task 772f5571-dda8-45d9-a616-781f807370cf is finished
2021-04-08 11:31:22,490 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(690b1dba-8825-47c2-b62e-94cfe2cfbac4, queue=awx_private_queue)
[2021-04-08 11:31:22,490: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(690b1dba-8825-47c2-b62e-94cfe2cfbac4, queue=awx_private_queue)
2021-04-08 11:31:22,521 DEBUG awx.main.dispatch delivered 690b1dba-8825-47c2-b62e-94cfe2cfbac4 to worker[192] qsize 0
2021-04-08 11:31:22,530 DEBUG awx.main.dispatch task 690b1dba-8825-47c2-b62e-94cfe2cfbac4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:31:22,547 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:31:22,549 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:31:22,553 DEBUG awx.main.dispatch task 690b1dba-8825-47c2-b62e-94cfe2cfbac4 is finished
2021-04-08 11:31:32,176 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(05e884af-eb68-4981-afa9-d74eb2fbfeb6, queue=dev-awx-0)
[2021-04-08 11:31:32,176: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(05e884af-eb68-4981-afa9-d74eb2fbfeb6, queue=dev-awx-0)
2021-04-08 11:31:32,204 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-08 11:31:32,250 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(238ee59b-393e-4d5e-8916-1e507b4efa10, queue=awx_private_queue)
[2021-04-08 11:31:32,250: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(238ee59b-393e-4d5e-8916-1e507b4efa10, queue=awx_private_queue)
2021-04-08 11:31:32,278 DEBUG awx.main.dispatch delivered 238ee59b-393e-4d5e-8916-1e507b4efa10 to worker[192] qsize 0
2021-04-08 11:31:32,284 DEBUG awx.main.dispatch task 238ee59b-393e-4d5e-8916-1e507b4efa10 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:31:32,296 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:31:32,297 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:31:32,301 DEBUG awx.main.dispatch task 238ee59b-393e-4d5e-8916-1e507b4efa10 is finished
2021-04-08 11:31:52,258 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(92805003-1cb8-4eca-bce2-2517d3103ed1, queue=awx_private_queue)
[2021-04-08 11:31:52,258: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(92805003-1cb8-4eca-bce2-2517d3103ed1, queue=awx_private_queue)
2021-04-08 11:31:52,297 DEBUG awx.main.dispatch delivered 92805003-1cb8-4eca-bce2-2517d3103ed1 to worker[195] qsize 0
2021-04-08 11:31:52,308 DEBUG awx.main.dispatch task 92805003-1cb8-4eca-bce2-2517d3103ed1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:31:52,329 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:31:52,331 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:31:52,335 DEBUG awx.main.dispatch task 92805003-1cb8-4eca-bce2-2517d3103ed1 is finished
2021-04-08 11:31:52,429 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(7ec37c9c-0223-48e4-b6a0-e155b885d5ab, queue=dev-awx-0)
[2021-04-08 11:31:52,429: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(7ec37c9c-0223-48e4-b6a0-e155b885d5ab, queue=dev-awx-0)
2021-04-08 11:31:52,470 DEBUG awx.main.dispatch delivered 7ec37c9c-0223-48e4-b6a0-e155b885d5ab to worker[179] qsize 0
2021-04-08 11:31:52,488 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(09e65466-fa80-41cc-b75e-8cf518eafdca, queue=awx_private_queue)
[2021-04-08 11:31:52,488: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(09e65466-fa80-41cc-b75e-8cf518eafdca, queue=awx_private_queue)
2021-04-08 11:31:52,477 DEBUG awx.main.dispatch task 7ec37c9c-0223-48e4-b6a0-e155b885d5ab starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:31:52,501 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139801040410176_10" does not exist
2021-04-08 11:31:52,506 DEBUG awx.main.dispatch task 7ec37c9c-0223-48e4-b6a0-e155b885d5ab is finished
2021-04-08 11:31:52,515 DEBUG awx.main.dispatch delivered 09e65466-fa80-41cc-b75e-8cf518eafdca to worker[179] qsize 0
2021-04-08 11:31:52,523 DEBUG awx.main.dispatch task 09e65466-fa80-41cc-b75e-8cf518eafdca starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:31:52,543 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:31:52,545 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:31:52,549 DEBUG awx.main.dispatch task 09e65466-fa80-41cc-b75e-8cf518eafdca is finished
RESULT 2
OKREADY
2021-04-08 11:32:12,276 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(16d95d3b-6eca-4948-9652-bcd65a7720c8, queue=awx_private_queue)
[2021-04-08 11:32:12,276: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(16d95d3b-6eca-4948-9652-bcd65a7720c8, queue=awx_private_queue)
2021-04-08 11:32:12,322 DEBUG awx.main.dispatch delivered 16d95d3b-6eca-4948-9652-bcd65a7720c8 to worker[179] qsize 0
2021-04-08 11:32:12,339 DEBUG awx.main.dispatch task 16d95d3b-6eca-4948-9652-bcd65a7720c8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:32:12,366 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:32:12,368 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:32:12,375 DEBUG awx.main.dispatch task 16d95d3b-6eca-4948-9652-bcd65a7720c8 is finished
2021-04-08 11:32:22,490 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(244327a5-0c2a-4f5a-b610-204e4be03373, queue=awx_private_queue)
[2021-04-08 11:32:22,490: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(244327a5-0c2a-4f5a-b610-204e4be03373, queue=awx_private_queue)
2021-04-08 11:32:22,519 DEBUG awx.main.dispatch delivered 244327a5-0c2a-4f5a-b610-204e4be03373 to worker[195] qsize 0
2021-04-08 11:32:22,528 DEBUG awx.main.dispatch task 244327a5-0c2a-4f5a-b610-204e4be03373 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:32:22,550 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:32:22,553 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:32:22,561 DEBUG awx.main.dispatch task 244327a5-0c2a-4f5a-b610-204e4be03373 is finished
2021-04-08 11:32:32,177 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e7007c0a-6374-4cfa-ab82-97d89b1c294d, queue=dev-awx-0)
[2021-04-08 11:32:32,177: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e7007c0a-6374-4cfa-ab82-97d89b1c294d, queue=dev-awx-0)
2021-04-08 11:32:32,219 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-08 11:32:32,270 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(231be3dd-24dc-4529-9561-f7ecfbf78053, queue=awx_private_queue)
[2021-04-08 11:32:32,270: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(231be3dd-24dc-4529-9561-f7ecfbf78053, queue=awx_private_queue)
2021-04-08 11:32:32,291 DEBUG awx.main.dispatch delivered 231be3dd-24dc-4529-9561-f7ecfbf78053 to worker[195] qsize 0
2021-04-08 11:32:32,297 DEBUG awx.main.dispatch task 231be3dd-24dc-4529-9561-f7ecfbf78053 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:32:32,308 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:32:32,310 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:32:32,313 DEBUG awx.main.dispatch task 231be3dd-24dc-4529-9561-f7ecfbf78053 is finished
2021-04-08 11:32:52,278 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e29a9ee4-2e91-4e68-9c7e-fa8d807dc623, queue=awx_private_queue)
[2021-04-08 11:32:52,278: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e29a9ee4-2e91-4e68-9c7e-fa8d807dc623, queue=awx_private_queue)
2021-04-08 11:32:52,303 DEBUG awx.main.dispatch delivered e29a9ee4-2e91-4e68-9c7e-fa8d807dc623 to worker[197] qsize 0
2021-04-08 11:32:52,313 DEBUG awx.main.dispatch task e29a9ee4-2e91-4e68-9c7e-fa8d807dc623 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:32:52,339 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:32:52,340 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:32:52,344 DEBUG awx.main.dispatch task e29a9ee4-2e91-4e68-9c7e-fa8d807dc623 is finished
2021-04-08 11:32:52,428 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(bfa52bab-a687-48d0-a374-9d1e9706640c, queue=dev-awx-0)
[2021-04-08 11:32:52,428: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(bfa52bab-a687-48d0-a374-9d1e9706640c, queue=dev-awx-0)
2021-04-08 11:32:52,457 DEBUG awx.main.dispatch delivered bfa52bab-a687-48d0-a374-9d1e9706640c to worker[197] qsize 0
2021-04-08 11:32:52,467 DEBUG awx.main.dispatch task bfa52bab-a687-48d0-a374-9d1e9706640c starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:32:52,482 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139801040410176_1" does not exist
2021-04-08 11:32:52,487 DEBUG awx.main.dispatch task bfa52bab-a687-48d0-a374-9d1e9706640c is finished
2021-04-08 11:32:52,488 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(53c7903c-57dd-40e0-a49f-50b1dc6d3b8d, queue=awx_private_queue)
[2021-04-08 11:32:52,488: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(53c7903c-57dd-40e0-a49f-50b1dc6d3b8d, queue=awx_private_queue)
2021-04-08 11:32:52,525 DEBUG awx.main.dispatch delivered 53c7903c-57dd-40e0-a49f-50b1dc6d3b8d to worker[197] qsize 0
2021-04-08 11:32:52,533 DEBUG awx.main.dispatch task 53c7903c-57dd-40e0-a49f-50b1dc6d3b8d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:32:52,544 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:32:52,547 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:32:52,552 DEBUG awx.main.dispatch task 53c7903c-57dd-40e0-a49f-50b1dc6d3b8d is finished
RESULT 2
OKREADY
2021-04-08 11:33:12,298 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(38ac7671-7571-4667-9a01-9fd69444a961, queue=awx_private_queue)
[2021-04-08 11:33:12,298: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(38ac7671-7571-4667-9a01-9fd69444a961, queue=awx_private_queue)
2021-04-08 11:33:12,327 DEBUG awx.main.dispatch delivered 38ac7671-7571-4667-9a01-9fd69444a961 to worker[192] qsize 0
2021-04-08 11:33:12,342 DEBUG awx.main.dispatch task 38ac7671-7571-4667-9a01-9fd69444a961 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:33:12,366 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:33:12,368 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:33:12,373 DEBUG awx.main.dispatch task 38ac7671-7571-4667-9a01-9fd69444a961 is finished
2021-04-08 11:33:22,496 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dfebedf1-fe39-4874-bc50-9ecab70d927c, queue=awx_private_queue)
[2021-04-08 11:33:22,496: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dfebedf1-fe39-4874-bc50-9ecab70d927c, queue=awx_private_queue)
2021-04-08 11:33:22,538 DEBUG awx.main.dispatch delivered dfebedf1-fe39-4874-bc50-9ecab70d927c to worker[195] qsize 0
2021-04-08 11:33:22,550 DEBUG awx.main.dispatch task dfebedf1-fe39-4874-bc50-9ecab70d927c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:33:22,577 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:33:22,580 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:33:22,585 DEBUG awx.main.dispatch task dfebedf1-fe39-4874-bc50-9ecab70d927c is finished
2021-04-08 11:33:32,178 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c68b4842-4eec-4537-82c3-2b3688e1350b, queue=dev-awx-0)
[2021-04-08 11:33:32,178: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c68b4842-4eec-4537-82c3-2b3688e1350b, queue=dev-awx-0)
2021-04-08 11:33:32,208 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-08 11:33:32,297 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(17316ade-604f-477f-8d7f-e15ffbedfe10, queue=awx_private_queue)
[2021-04-08 11:33:32,297: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(17316ade-604f-477f-8d7f-e15ffbedfe10, queue=awx_private_queue)
2021-04-08 11:33:32,325 DEBUG awx.main.dispatch delivered 17316ade-604f-477f-8d7f-e15ffbedfe10 to worker[192] qsize 0
2021-04-08 11:33:32,339 DEBUG awx.main.dispatch task 17316ade-604f-477f-8d7f-e15ffbedfe10 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:33:32,355 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:33:32,358 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:33:32,372 DEBUG awx.main.dispatch task 17316ade-604f-477f-8d7f-e15ffbedfe10 is finished
2021-04-08 11:33:52,308 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f859b008-61c5-4fb3-bf0b-077e7808c4e3, queue=awx_private_queue)
[2021-04-08 11:33:52,308: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f859b008-61c5-4fb3-bf0b-077e7808c4e3, queue=awx_private_queue)
2021-04-08 11:33:52,345 DEBUG awx.main.dispatch delivered f859b008-61c5-4fb3-bf0b-077e7808c4e3 to worker[192] qsize 0
2021-04-08 11:33:52,354 DEBUG awx.main.dispatch task f859b008-61c5-4fb3-bf0b-077e7808c4e3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:33:52,366 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:33:52,367 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:33:52,371 DEBUG awx.main.dispatch task f859b008-61c5-4fb3-bf0b-077e7808c4e3 is finished
2021-04-08 11:33:52,429 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(772dde66-995d-4271-b379-23868e5005b6, queue=dev-awx-0)
[2021-04-08 11:33:52,429: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(772dde66-995d-4271-b379-23868e5005b6, queue=dev-awx-0)
2021-04-08 11:33:52,453 DEBUG awx.main.dispatch delivered 772dde66-995d-4271-b379-23868e5005b6 to worker[192] qsize 0
2021-04-08 11:33:52,461 DEBUG awx.main.dispatch task 772dde66-995d-4271-b379-23868e5005b6 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:33:52,472 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139801040410176_5" does not exist
2021-04-08 11:33:52,476 DEBUG awx.main.dispatch task 772dde66-995d-4271-b379-23868e5005b6 is finished
2021-04-08 11:33:52,489 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(809a8eb8-f94b-4d85-a380-0689085eb058, queue=awx_private_queue)
[2021-04-08 11:33:52,489: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(809a8eb8-f94b-4d85-a380-0689085eb058, queue=awx_private_queue)
2021-04-08 11:33:52,511 DEBUG awx.main.dispatch delivered 809a8eb8-f94b-4d85-a380-0689085eb058 to worker[179] qsize 0
2021-04-08 11:33:52,519 DEBUG awx.main.dispatch task 809a8eb8-f94b-4d85-a380-0689085eb058 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:33:52,540 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:33:52,542 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:33:52,547 DEBUG awx.main.dispatch task 809a8eb8-f94b-4d85-a380-0689085eb058 is finished
RESULT 2
OKREADY
2021-04-08 11:34:12,312 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(af75242f-e1d1-43da-8a4f-2bd138ac34ab, queue=awx_private_queue)
[2021-04-08 11:34:12,312: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(af75242f-e1d1-43da-8a4f-2bd138ac34ab, queue=awx_private_queue)
2021-04-08 11:34:12,335 DEBUG awx.main.dispatch delivered af75242f-e1d1-43da-8a4f-2bd138ac34ab to worker[179] qsize 0
2021-04-08 11:34:12,344 DEBUG awx.main.dispatch task af75242f-e1d1-43da-8a4f-2bd138ac34ab starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:34:12,357 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:34:12,358 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:34:12,362 DEBUG awx.main.dispatch task af75242f-e1d1-43da-8a4f-2bd138ac34ab is finished
2021-04-08 11:34:22,490 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d51194ce-8292-4309-87ff-9bf87693be3a, queue=awx_private_queue)
[2021-04-08 11:34:22,490: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d51194ce-8292-4309-87ff-9bf87693be3a, queue=awx_private_queue)
2021-04-08 11:34:22,516 DEBUG awx.main.dispatch delivered d51194ce-8292-4309-87ff-9bf87693be3a to worker[179] qsize 0
2021-04-08 11:34:22,522 DEBUG awx.main.dispatch task d51194ce-8292-4309-87ff-9bf87693be3a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:34:22,535 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:34:22,537 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:34:22,542 DEBUG awx.main.dispatch task d51194ce-8292-4309-87ff-9bf87693be3a is finished
2021-04-08 11:34:32,176 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5c7f4692-1d04-454a-ba58-94f573523b81, queue=dev-awx-0)
[2021-04-08 11:34:32,176: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5c7f4692-1d04-454a-ba58-94f573523b81, queue=dev-awx-0)
2021-04-08 11:34:32,202 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-08 11:34:32,313 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(62b0afa0-41fc-4d0e-b19b-654fd9ebc243, queue=awx_private_queue)
[2021-04-08 11:34:32,313: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(62b0afa0-41fc-4d0e-b19b-654fd9ebc243, queue=awx_private_queue)
2021-04-08 11:34:32,340 DEBUG awx.main.dispatch delivered 62b0afa0-41fc-4d0e-b19b-654fd9ebc243 to worker[179] qsize 0
2021-04-08 11:34:32,346 DEBUG awx.main.dispatch task 62b0afa0-41fc-4d0e-b19b-654fd9ebc243 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:34:32,358 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:34:32,359 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:34:32,363 DEBUG awx.main.dispatch task 62b0afa0-41fc-4d0e-b19b-654fd9ebc243 is finished
2021-04-08 11:34:52,331 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(13ffc307-22ac-49a5-a437-8da0f3f4604e, queue=awx_private_queue)
[2021-04-08 11:34:52,331: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(13ffc307-22ac-49a5-a437-8da0f3f4604e, queue=awx_private_queue)
2021-04-08 11:34:52,379 DEBUG awx.main.dispatch delivered 13ffc307-22ac-49a5-a437-8da0f3f4604e to worker[179] qsize 0
2021-04-08 11:34:52,392 DEBUG awx.main.dispatch task 13ffc307-22ac-49a5-a437-8da0f3f4604e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:34:52,408 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:34:52,410 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:34:52,415 DEBUG awx.main.dispatch task 13ffc307-22ac-49a5-a437-8da0f3f4604e is finished
2021-04-08 11:34:52,429 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a1b2044b-c4ba-4c10-b38a-66a36386dc53, queue=dev-awx-0)
[2021-04-08 11:34:52,429: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a1b2044b-c4ba-4c10-b38a-66a36386dc53, queue=dev-awx-0)
2021-04-08 11:34:52,460 DEBUG awx.main.dispatch delivered a1b2044b-c4ba-4c10-b38a-66a36386dc53 to worker[192] qsize 0
2021-04-08 11:34:52,468 DEBUG awx.main.dispatch task a1b2044b-c4ba-4c10-b38a-66a36386dc53 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:34:52,491 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139801040410176_6" does not exist
2021-04-08 11:34:52,489 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1e41aa6d-4014-42dc-930c-04dab93c5dea, queue=awx_private_queue)
[2021-04-08 11:34:52,489: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1e41aa6d-4014-42dc-930c-04dab93c5dea, queue=awx_private_queue)
2021-04-08 11:34:52,497 DEBUG awx.main.dispatch task a1b2044b-c4ba-4c10-b38a-66a36386dc53 is finished
2021-04-08 11:34:52,514 DEBUG awx.main.dispatch delivered 1e41aa6d-4014-42dc-930c-04dab93c5dea to worker[195] qsize 0
2021-04-08 11:34:52,525 DEBUG awx.main.dispatch task 1e41aa6d-4014-42dc-930c-04dab93c5dea starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:34:52,548 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:34:52,550 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:34:52,554 DEBUG awx.main.dispatch task 1e41aa6d-4014-42dc-930c-04dab93c5dea is finished
RESULT 2
OKREADY
2021-04-08 11:35:12,331 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(79ce0d5e-e846-4726-8fea-46d235124335, queue=awx_private_queue)
[2021-04-08 11:35:12,331: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(79ce0d5e-e846-4726-8fea-46d235124335, queue=awx_private_queue)
2021-04-08 11:35:12,359 DEBUG awx.main.dispatch delivered 79ce0d5e-e846-4726-8fea-46d235124335 to worker[195] qsize 0
2021-04-08 11:35:12,368 DEBUG awx.main.dispatch task 79ce0d5e-e846-4726-8fea-46d235124335 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:35:12,381 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:35:12,382 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:35:12,386 DEBUG awx.main.dispatch task 79ce0d5e-e846-4726-8fea-46d235124335 is finished
2021-04-08 11:35:22,490 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d3908d35-c4ce-4bc6-817c-bffc5076e68a, queue=awx_private_queue)
[2021-04-08 11:35:22,490: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d3908d35-c4ce-4bc6-817c-bffc5076e68a, queue=awx_private_queue)
2021-04-08 11:35:22,524 DEBUG awx.main.dispatch delivered d3908d35-c4ce-4bc6-817c-bffc5076e68a to worker[179] qsize 0
2021-04-08 11:35:22,531 DEBUG awx.main.dispatch task d3908d35-c4ce-4bc6-817c-bffc5076e68a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:35:22,549 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:35:22,551 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:35:22,554 DEBUG awx.main.dispatch task d3908d35-c4ce-4bc6-817c-bffc5076e68a is finished
2021-04-08 11:35:32,176 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c858a1fd-1bd4-43d1-9a43-65380d4a722d, queue=dev-awx-0)
[2021-04-08 11:35:32,176: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c858a1fd-1bd4-43d1-9a43-65380d4a722d, queue=dev-awx-0)
2021-04-08 11:35:32,207 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-08 11:35:32,330 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8f1f32ae-2ab7-4f39-a1f7-62abf1ba7905, queue=awx_private_queue)
[2021-04-08 11:35:32,330: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8f1f32ae-2ab7-4f39-a1f7-62abf1ba7905, queue=awx_private_queue)
2021-04-08 11:35:32,351 DEBUG awx.main.dispatch delivered 8f1f32ae-2ab7-4f39-a1f7-62abf1ba7905 to worker[197] qsize 0
2021-04-08 11:35:32,362 DEBUG awx.main.dispatch task 8f1f32ae-2ab7-4f39-a1f7-62abf1ba7905 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:35:32,390 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:35:32,392 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:35:32,398 DEBUG awx.main.dispatch task 8f1f32ae-2ab7-4f39-a1f7-62abf1ba7905 is finished
2021-04-08 11:35:52,337 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d9f50644-1382-4c5e-90c9-09d53882c3e2, queue=awx_private_queue)
[2021-04-08 11:35:52,337: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d9f50644-1382-4c5e-90c9-09d53882c3e2, queue=awx_private_queue)
2021-04-08 11:35:52,378 DEBUG awx.main.dispatch delivered d9f50644-1382-4c5e-90c9-09d53882c3e2 to worker[195] qsize 0
2021-04-08 11:35:52,393 DEBUG awx.main.dispatch task d9f50644-1382-4c5e-90c9-09d53882c3e2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:35:52,414 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:35:52,416 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:35:52,428 DEBUG awx.main.dispatch task d9f50644-1382-4c5e-90c9-09d53882c3e2 is finished
2021-04-08 11:35:52,428 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(9583ede8-546a-4128-8a14-47577c420a55, queue=dev-awx-0)
[2021-04-08 11:35:52,428: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(9583ede8-546a-4128-8a14-47577c420a55, queue=dev-awx-0)
2021-04-08 11:35:52,450 DEBUG awx.main.dispatch delivered 9583ede8-546a-4128-8a14-47577c420a55 to worker[197] qsize 0
2021-04-08 11:35:52,457 DEBUG awx.main.dispatch task 9583ede8-546a-4128-8a14-47577c420a55 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:35:52,474 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139801040410176_2" does not exist
2021-04-08 11:35:52,478 DEBUG awx.main.dispatch task 9583ede8-546a-4128-8a14-47577c420a55 is finished
2021-04-08 11:35:52,489 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e937f3ee-de9f-4506-9331-2eed07a5c0b5, queue=awx_private_queue)
[2021-04-08 11:35:52,489: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e937f3ee-de9f-4506-9331-2eed07a5c0b5, queue=awx_private_queue)
2021-04-08 11:35:52,509 DEBUG awx.main.dispatch delivered e937f3ee-de9f-4506-9331-2eed07a5c0b5 to worker[195] qsize 0
2021-04-08 11:35:52,514 DEBUG awx.main.dispatch task e937f3ee-de9f-4506-9331-2eed07a5c0b5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:35:52,525 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:35:52,527 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:35:52,538 DEBUG awx.main.dispatch task e937f3ee-de9f-4506-9331-2eed07a5c0b5 is finished
RESULT 2
OKREADY
2021-04-08 11:36:12,348 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9f291f8b-aa8e-4ca3-8f70-3403bb672ff6, queue=awx_private_queue)
[2021-04-08 11:36:12,348: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9f291f8b-aa8e-4ca3-8f70-3403bb672ff6, queue=awx_private_queue)
2021-04-08 11:36:12,378 DEBUG awx.main.dispatch delivered 9f291f8b-aa8e-4ca3-8f70-3403bb672ff6 to worker[197] qsize 0
2021-04-08 11:36:12,390 DEBUG awx.main.dispatch task 9f291f8b-aa8e-4ca3-8f70-3403bb672ff6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:36:12,404 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:36:12,406 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:36:12,411 DEBUG awx.main.dispatch task 9f291f8b-aa8e-4ca3-8f70-3403bb672ff6 is finished
2021-04-08 11:36:22,491 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(053f952d-fcc2-4788-9760-8bd8ed80aebf, queue=awx_private_queue)
[2021-04-08 11:36:22,491: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(053f952d-fcc2-4788-9760-8bd8ed80aebf, queue=awx_private_queue)
2021-04-08 11:36:22,522 DEBUG awx.main.dispatch delivered 053f952d-fcc2-4788-9760-8bd8ed80aebf to worker[192] qsize 0
2021-04-08 11:36:22,531 DEBUG awx.main.dispatch task 053f952d-fcc2-4788-9760-8bd8ed80aebf starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:36:22,553 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:36:22,555 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:36:22,559 DEBUG awx.main.dispatch task 053f952d-fcc2-4788-9760-8bd8ed80aebf is finished
2021-04-08 11:36:32,178 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(9e8ffa0a-ab09-417b-b7b2-51f8258deb30, queue=dev-awx-0)
[2021-04-08 11:36:32,178: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(9e8ffa0a-ab09-417b-b7b2-51f8258deb30, queue=dev-awx-0)
2021-04-08 11:36:32,218 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-08 11:36:32,347 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(438502b2-4294-47af-bbc6-efca6297f5c6, queue=awx_private_queue)
[2021-04-08 11:36:32,347: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(438502b2-4294-47af-bbc6-efca6297f5c6, queue=awx_private_queue)
2021-04-08 11:36:32,380 DEBUG awx.main.dispatch delivered 438502b2-4294-47af-bbc6-efca6297f5c6 to worker[179] qsize 0
2021-04-08 11:36:32,390 DEBUG awx.main.dispatch task 438502b2-4294-47af-bbc6-efca6297f5c6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:36:32,416 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:36:32,418 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:36:32,424 DEBUG awx.main.dispatch task 438502b2-4294-47af-bbc6-efca6297f5c6 is finished
2021-04-08 11:36:52,355 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(190d529b-e39e-42a3-a68d-8701741986d0, queue=awx_private_queue)
[2021-04-08 11:36:52,355: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(190d529b-e39e-42a3-a68d-8701741986d0, queue=awx_private_queue)
2021-04-08 11:36:52,383 DEBUG awx.main.dispatch delivered 190d529b-e39e-42a3-a68d-8701741986d0 to worker[192] qsize 0
2021-04-08 11:36:52,390 DEBUG awx.main.dispatch task 190d529b-e39e-42a3-a68d-8701741986d0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:36:52,401 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:36:52,402 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:36:52,407 DEBUG awx.main.dispatch task 190d529b-e39e-42a3-a68d-8701741986d0 is finished
2021-04-08 11:36:52,429 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b84c0e4c-ff4b-4d02-8acf-4c266b1ce085, queue=dev-awx-0)
[2021-04-08 11:36:52,429: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b84c0e4c-ff4b-4d02-8acf-4c266b1ce085, queue=dev-awx-0)
2021-04-08 11:36:52,452 DEBUG awx.main.dispatch delivered b84c0e4c-ff4b-4d02-8acf-4c266b1ce085 to worker[192] qsize 0
2021-04-08 11:36:52,457 DEBUG awx.main.dispatch task b84c0e4c-ff4b-4d02-8acf-4c266b1ce085 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:36:52,469 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139801040410176_7" does not exist
2021-04-08 11:36:52,472 DEBUG awx.main.dispatch task b84c0e4c-ff4b-4d02-8acf-4c266b1ce085 is finished
2021-04-08 11:36:52,489 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ce7420b1-766f-4f95-bd32-f274f4063a75, queue=awx_private_queue)
[2021-04-08 11:36:52,489: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ce7420b1-766f-4f95-bd32-f274f4063a75, queue=awx_private_queue)
2021-04-08 11:36:52,520 DEBUG awx.main.dispatch delivered ce7420b1-766f-4f95-bd32-f274f4063a75 to worker[179] qsize 0
2021-04-08 11:36:52,526 DEBUG awx.main.dispatch task ce7420b1-766f-4f95-bd32-f274f4063a75 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:36:52,537 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:36:52,539 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:36:52,543 DEBUG awx.main.dispatch task ce7420b1-766f-4f95-bd32-f274f4063a75 is finished
RESULT 2
OKREADY
2021-04-08 11:37:12,369 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(10093950-c2c3-49b1-9804-476c7ee1dc36, queue=awx_private_queue)
[2021-04-08 11:37:12,369: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(10093950-c2c3-49b1-9804-476c7ee1dc36, queue=awx_private_queue)
2021-04-08 11:37:12,409 DEBUG awx.main.dispatch delivered 10093950-c2c3-49b1-9804-476c7ee1dc36 to worker[195] qsize 0
2021-04-08 11:37:12,424 DEBUG awx.main.dispatch task 10093950-c2c3-49b1-9804-476c7ee1dc36 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:37:12,445 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:37:12,447 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:37:12,452 DEBUG awx.main.dispatch task 10093950-c2c3-49b1-9804-476c7ee1dc36 is finished
2021-04-08 11:37:13,734 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(78fbd5aa-0a65-475b-9238-36d7f7aaf11e, queue=dev-awx-0)
[2021-04-08 11:37:13,734: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(78fbd5aa-0a65-475b-9238-36d7f7aaf11e, queue=dev-awx-0)
2021-04-08 11:37:13,778 DEBUG awx.main.dispatch delivered 78fbd5aa-0a65-475b-9238-36d7f7aaf11e to worker[195] qsize 0
2021-04-08 11:37:13,786 DEBUG awx.main.dispatch task 78fbd5aa-0a65-475b-9238-36d7f7aaf11e starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-04-08 11:37:13,799 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-04-08 11:37:13,805 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 470, in awx_isolated_heartbeat
for isolated_instance in isolated_instance_qs:
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-04-08 11:37:13,810 DEBUG awx.main.dispatch task 78fbd5aa-0a65-475b-9238-36d7f7aaf11e is finished
2021-04-08 11:37:22,490 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(973a6dac-89b3-43e9-b87e-d8809f73a6b7, queue=awx_private_queue)
[2021-04-08 11:37:22,490: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(973a6dac-89b3-43e9-b87e-d8809f73a6b7, queue=awx_private_queue)
2021-04-08 11:37:22,514 DEBUG awx.main.dispatch delivered 973a6dac-89b3-43e9-b87e-d8809f73a6b7 to worker[197] qsize 0
2021-04-08 11:37:22,522 DEBUG awx.main.dispatch task 973a6dac-89b3-43e9-b87e-d8809f73a6b7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:37:22,542 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:37:22,545 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:37:22,549 DEBUG awx.main.dispatch task 973a6dac-89b3-43e9-b87e-d8809f73a6b7 is finished
2021-04-08 11:37:32,177 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c5aeb1ac-5dc9-4432-b33b-a146d6c671a0, queue=dev-awx-0)
[2021-04-08 11:37:32,177: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c5aeb1ac-5dc9-4432-b33b-a146d6c671a0, queue=dev-awx-0)
2021-04-08 11:37:32,207 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-08 11:37:32,364 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(62e5c95c-0462-4665-a6fd-8f90b44ed0f5, queue=awx_private_queue)
[2021-04-08 11:37:32,364: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(62e5c95c-0462-4665-a6fd-8f90b44ed0f5, queue=awx_private_queue)
2021-04-08 11:37:32,392 DEBUG awx.main.dispatch delivered 62e5c95c-0462-4665-a6fd-8f90b44ed0f5 to worker[179] qsize 0
2021-04-08 11:37:32,402 DEBUG awx.main.dispatch task 62e5c95c-0462-4665-a6fd-8f90b44ed0f5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:37:32,423 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:37:32,425 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:37:32,432 DEBUG awx.main.dispatch task 62e5c95c-0462-4665-a6fd-8f90b44ed0f5 is finished
2021-04-08 11:37:52,371 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(eae65578-8159-4d6c-a873-1b4e8f966130, queue=awx_private_queue)
[2021-04-08 11:37:52,371: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(eae65578-8159-4d6c-a873-1b4e8f966130, queue=awx_private_queue)
2021-04-08 11:37:52,403 DEBUG awx.main.dispatch delivered eae65578-8159-4d6c-a873-1b4e8f966130 to worker[179] qsize 0
2021-04-08 11:37:52,411 DEBUG awx.main.dispatch task eae65578-8159-4d6c-a873-1b4e8f966130 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:37:52,422 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:37:52,423 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:37:52,428 DEBUG awx.main.dispatch task eae65578-8159-4d6c-a873-1b4e8f966130 is finished
2021-04-08 11:37:52,429 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b9d1d2a4-eb38-4aff-9515-755e4ed5a30f, queue=dev-awx-0)
[2021-04-08 11:37:52,429: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b9d1d2a4-eb38-4aff-9515-755e4ed5a30f, queue=dev-awx-0)
2021-04-08 11:37:52,450 DEBUG awx.main.dispatch delivered b9d1d2a4-eb38-4aff-9515-755e4ed5a30f to worker[179] qsize 0
2021-04-08 11:37:52,458 DEBUG awx.main.dispatch task b9d1d2a4-eb38-4aff-9515-755e4ed5a30f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-08 11:37:52,470 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139801040410176_11" does not exist
2021-04-08 11:37:52,479 DEBUG awx.main.dispatch task b9d1d2a4-eb38-4aff-9515-755e4ed5a30f is finished
2021-04-08 11:37:52,489 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f2157b1b-9b31-4ef9-8446-51ad37c5cbe9, queue=awx_private_queue)
[2021-04-08 11:37:52,489: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f2157b1b-9b31-4ef9-8446-51ad37c5cbe9, queue=awx_private_queue)
2021-04-08 11:37:52,524 DEBUG awx.main.dispatch delivered f2157b1b-9b31-4ef9-8446-51ad37c5cbe9 to worker[192] qsize 0
2021-04-08 11:37:52,534 DEBUG awx.main.dispatch task f2157b1b-9b31-4ef9-8446-51ad37c5cbe9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:37:52,558 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:37:52,561 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:37:52,567 DEBUG awx.main.dispatch task f2157b1b-9b31-4ef9-8446-51ad37c5cbe9 is finished
RESULT 2
OKREADY
2021-04-08 11:38:12,375 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e0785ef8-e642-44b7-97dc-e8b2865ffbab, queue=awx_private_queue)
[2021-04-08 11:38:12,375: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e0785ef8-e642-44b7-97dc-e8b2865ffbab, queue=awx_private_queue)
2021-04-08 11:38:12,418 DEBUG awx.main.dispatch delivered e0785ef8-e642-44b7-97dc-e8b2865ffbab to worker[192] qsize 0
2021-04-08 11:38:12,432 DEBUG awx.main.dispatch task e0785ef8-e642-44b7-97dc-e8b2865ffbab starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:38:12,443 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:38:12,445 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-08 11:38:12,448 DEBUG awx.main.dispatch task e0785ef8-e642-44b7-97dc-e8b2865ffbab is finished
2021-04-08 11:38:22,491 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ac146073-5be7-422e-b8a6-07a85043ca17, queue=awx_private_queue)
[2021-04-08 11:38:22,491: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ac146073-5be7-422e-b8a6-07a85043ca17, queue=awx_private_queue)
2021-04-08 11:38:22,519 DEBUG awx.main.dispatch delivered ac146073-5be7-422e-b8a6-07a85043ca17 to worker[192] qsize 0
2021-04-08 11:38:22,528 DEBUG awx.main.dispatch task ac146073-5be7-422e-b8a6-07a85043ca17 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-08 11:38:22,544 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-08 11:38:22,547 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-08 11:38:22,553 DEBUG awx.main.dispatch task ac146073-5be7-422e-b8a6-07a85043ca17 is finished
2021-04-08 11:38:32,178 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(425748fd-5703-4b4c-8155-397e393fbf1c, queue=dev-awx-0)
[2021-04-08 11:38:32,178: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(425748fd-5703-4b4c-8155-397e393fbf1c, queue=dev-awx-0)
2021-04-08 11:38:32,213 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-08 11:38:32,371 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b9eef0d0-866f-44b0-8bd0-c6a5a6d6cf47, queue=awx_private_queue)
[2021-04-08 11:38:32,371: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b9eef0d0-866f-44b0-8bd0-c6a5a6d6cf47, queue=awx_private_queue)
2021-04-08 11:38:32,402 DEBUG awx.main.dispatch delivered b9eef0d0-866f-44b0-8bd0-c6a5a6d6cf47 to worker[192] qsize 0
2021-04-08 11:38:32,410 DEBUG awx.main.dispatch task b9eef0d0-866f-44b0-8bd0-c6a5a6d6cf47 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-08 11:38:32,424 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-08 11:38:32,426 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...