By type
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"elapsed": 0,
"match_groupdict": {},
"match_groups": [],
"path": null,
"port": 5432,
"search_regex": null,
"state": "started"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"db": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
}
Default organization added.
Demo Credential, Inventory, and Job Template added.
(changed: True)
2021-06-06 22:48:02,443 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(65483c12-2ef8-4d3d-8580-c210811102f8, queue=awx_private_queue)
Successfully registered instance dev-awx-0
(changed: True)
2021-06-06 22:48:06,115 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(2c8b62a2-b3d0-4741-9511-882ae480d721, queue=awx_private_queue)
2021-06-06 22:48:06,159 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(c5a8586b-64be-4156-bff4-4f7817fc0aa7, queue=awx_private_queue)
Creating instance group tower
(changed: True)
/usr/bin/launch_awx_task.sh: line 27: unset: `#!/bin/sh': not a valid identifier
2021-06-06 22:48:06,894 INFO RPC interface 'supervisor' initialized
2021-06-06 22:48:06,894 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2021-06-06 22:48:06,894 INFO supervisord started with pid 151
2021-06-06 22:48:07,899 INFO spawned: 'awx-config-watcher' with pid 154
2021-06-06 22:48:07,903 INFO spawned: 'dispatcher' with pid 155
2021-06-06 22:48:07,906 INFO spawned: 'callback-receiver' with pid 156
2021-06-06 22:48:07,911 INFO spawned: 'channels-worker' with pid 157
READY
2021-06-06 22:48:09,014 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-06-06 22:48:09,014 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-06-06 22:48:09,014 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-06-06 22:48:09,014 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-06-06 22:48:11,161 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-06-06 22:48:11,161 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-06-06 22:48:11,166 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2021-06-06 22:48:11,166 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2021-06-06 22:48:11,171 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2021-06-06 22:48:11,171 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2021-06-06 22:48:11,176 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2021-06-06 22:48:11,176 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2021-06-06 22:48:11,260 - INFO - runworker - Using single-threaded worker.
2021-06-06 22:48:11,260 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer)
2021-06-06 22:48:11,261 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive
2021-06-06 22:48:11,261 ERROR celery.beat Removing corrupted schedule file '/var/lib/awx/beat.db': error(11, 'Resource temporarily unavailable')
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 485, in setup_schedule
self._store = self._open_schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 475, in _open_schedule
return self.persistence.open(self.schedule_filename, writeback=True)
File "/usr/lib64/python3.6/shelve.py", line 243, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/usr/lib64/python3.6/shelve.py", line 227, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
File "/usr/lib64/python3.6/dbm/__init__.py", line 94, in open
return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable
2021-06-06 22:48:11,270 WARNING awx.main.dispatch scaling up worker pid:181
2021-06-06 22:48:11,279 WARNING awx.main.dispatch scaling up worker pid:182
2021-06-06 22:48:11,287 WARNING awx.main.dispatch scaling up worker pid:183
2021-06-06 22:48:11,295 WARNING awx.main.dispatch scaling up worker pid:184
2021-06-06 22:48:11,300 DEBUG awx.main.tasks Syncing Schedules
2021-06-06 22:48:11,541 DEBUG awx.main.tasks Waited 0.0010313987731933594 seconds to obtain lock name: cluster_policy_lock
2021-06-06 22:48:11,552 DEBUG awx.main.tasks Total non-isolated instances:1 available for policy: 1
2021-06-06 22:48:11,553 DEBUG awx.main.tasks Policy percentage, adding Instances [1] to Group tower
2021-06-06 22:48:11,553 DEBUG awx.main.tasks Adding instances [1] to group tower
2021-06-06 22:48:11,560 DEBUG awx.main.tasks Cluster policy computation finished in 0.01864004135131836 seconds
2021-06-06 22:48:11,561 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 22:48:11,598 DEBUG awx.main.dispatch listening on ['awx_private_queue [direct]', 'dev-awx-0 [direct]', 'tower_broadcast_all_dev-awx-0 [fanout]']
2021-06-06 22:48:31,395 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7c552a76-cd06-4a08-958e-a7656a0d29b3, queue=awx_private_queue)
[2021-06-06 22:48:31,395: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7c552a76-cd06-4a08-958e-a7656a0d29b3, queue=awx_private_queue)
2021-06-06 22:48:31,412 DEBUG awx.main.dispatch delivered 7c552a76-cd06-4a08-958e-a7656a0d29b3 to worker[183] qsize 0
2021-06-06 22:48:31,418 DEBUG awx.main.dispatch task 7c552a76-cd06-4a08-958e-a7656a0d29b3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:48:31,420 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:48:31,443 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:48:31,483 DEBUG awx.main.dispatch task 7c552a76-cd06-4a08-958e-a7656a0d29b3 is finished
2021-06-06 22:48:41,390 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5993495a-3414-427d-8dfa-ca9e1e48a50b, queue=awx_private_queue)
[2021-06-06 22:48:41,390: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5993495a-3414-427d-8dfa-ca9e1e48a50b, queue=awx_private_queue)
2021-06-06 22:48:41,424 DEBUG awx.main.dispatch delivered 5993495a-3414-427d-8dfa-ca9e1e48a50b to worker[184] qsize 0
2021-06-06 22:48:41,437 DEBUG awx.main.dispatch task 5993495a-3414-427d-8dfa-ca9e1e48a50b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:48:41,465 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:48:41,482 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:48:41.477254+00:00
2021-06-06 22:48:41,505 DEBUG awx.main.dispatch task 5993495a-3414-427d-8dfa-ca9e1e48a50b is finished
2021-06-06 22:48:51,384 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c1013726-d5f6-4228-801d-4ab417e65c93, queue=awx_private_queue)
[2021-06-06 22:48:51,384: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c1013726-d5f6-4228-801d-4ab417e65c93, queue=awx_private_queue)
2021-06-06 22:48:51,401 DEBUG awx.main.dispatch delivered c1013726-d5f6-4228-801d-4ab417e65c93 to worker[181] qsize 0
2021-06-06 22:48:51,407 DEBUG awx.main.dispatch task c1013726-d5f6-4228-801d-4ab417e65c93 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:48:51,412 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:48:51,435 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:48:51,473 DEBUG awx.main.dispatch task c1013726-d5f6-4228-801d-4ab417e65c93 is finished
RESULT 2
OKREADY
2021-06-06 22:49:11,393 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(0008c5cc-83cb-46ed-af96-3a6bedde8fc7, queue=dev-awx-0)
[2021-06-06 22:49:11,393: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(0008c5cc-83cb-46ed-af96-3a6bedde8fc7, queue=dev-awx-0)
2021-06-06 22:49:11,581 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ce778c91-97a4-4e37-889c-c2c569f8ed0d, queue=dev-awx-0)
[2021-06-06 22:49:11,581: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ce778c91-97a4-4e37-889c-c2c569f8ed0d, queue=dev-awx-0)
2021-06-06 22:49:11,594 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a8aaf58b-2a1d-4604-a5f7-32dbb20f5e1c, queue=awx_private_queue)
[2021-06-06 22:49:11,594: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a8aaf58b-2a1d-4604-a5f7-32dbb20f5e1c, queue=awx_private_queue)
2021-06-06 22:49:11,596 DEBUG awx.main.dispatch delivered 0008c5cc-83cb-46ed-af96-3a6bedde8fc7 to worker[184] qsize 0
2021-06-06 22:49:11,599 DEBUG awx.main.dispatch delivered ce778c91-97a4-4e37-889c-c2c569f8ed0d to worker[183] qsize 0
2021-06-06 22:49:11,599 DEBUG awx.main.dispatch task 0008c5cc-83cb-46ed-af96-3a6bedde8fc7 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 22:49:11,601 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 22:49:11,602 DEBUG awx.main.dispatch task ce778c91-97a4-4e37-889c-c2c569f8ed0d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 22:49:11,610 DEBUG awx.main.dispatch delivered a8aaf58b-2a1d-4604-a5f7-32dbb20f5e1c to worker[181] qsize 0
2021-06-06 22:49:11,612 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4d3c0f6a-7792-42b1-8bbc-d0b01260bf9a, queue=awx_private_queue)
[2021-06-06 22:49:11,612: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4d3c0f6a-7792-42b1-8bbc-d0b01260bf9a, queue=awx_private_queue)
2021-06-06 22:49:11,612 DEBUG awx.main.dispatch task a8aaf58b-2a1d-4604-a5f7-32dbb20f5e1c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:49:11,614 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:49:11,624 DEBUG awx.main.dispatch task ce778c91-97a4-4e37-889c-c2c569f8ed0d is finished
2021-06-06 22:49:11,630 DEBUG awx.main.dispatch delivered 4d3c0f6a-7792-42b1-8bbc-d0b01260bf9a to worker[182] qsize 0
2021-06-06 22:49:11,632 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:49:11,635 DEBUG awx.main.dispatch task 0008c5cc-83cb-46ed-af96-3a6bedde8fc7 is finished
2021-06-06 22:49:11,635 DEBUG awx.main.dispatch task 4d3c0f6a-7792-42b1-8bbc-d0b01260bf9a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:49:11,649 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:49:11,653 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:48:41.465439+00:00
2021-06-06 22:49:11,660 DEBUG awx.main.dispatch task a8aaf58b-2a1d-4604-a5f7-32dbb20f5e1c is finished
2021-06-06 22:49:11,668 DEBUG awx.main.dispatch task 4d3c0f6a-7792-42b1-8bbc-d0b01260bf9a is finished
2021-06-06 22:49:31,605 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9304fa2c-b806-47a6-9f7a-233c1ca3b87f, queue=awx_private_queue)
[2021-06-06 22:49:31,605: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9304fa2c-b806-47a6-9f7a-233c1ca3b87f, queue=awx_private_queue)
2021-06-06 22:49:31,619 DEBUG awx.main.dispatch delivered 9304fa2c-b806-47a6-9f7a-233c1ca3b87f to worker[184] qsize 0
2021-06-06 22:49:31,622 DEBUG awx.main.dispatch task 9304fa2c-b806-47a6-9f7a-233c1ca3b87f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:49:31,623 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:49:31,641 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:49:31,680 DEBUG awx.main.dispatch task 9304fa2c-b806-47a6-9f7a-233c1ca3b87f is finished
2021-06-06 22:49:41,612 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d02b262d-bb95-4b7c-9c39-171172587026, queue=awx_private_queue)
[2021-06-06 22:49:41,612: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d02b262d-bb95-4b7c-9c39-171172587026, queue=awx_private_queue)
2021-06-06 22:49:41,629 DEBUG awx.main.dispatch delivered d02b262d-bb95-4b7c-9c39-171172587026 to worker[182] qsize 0
2021-06-06 22:49:41,631 DEBUG awx.main.dispatch task d02b262d-bb95-4b7c-9c39-171172587026 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:49:41,640 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:49:41,643 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:49:11.649956+00:00
2021-06-06 22:49:41,659 DEBUG awx.main.dispatch task d02b262d-bb95-4b7c-9c39-171172587026 is finished
2021-06-06 22:49:51,605 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ab8f489c-79f3-4244-8067-7e18f1c1b0fe, queue=awx_private_queue)
[2021-06-06 22:49:51,605: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ab8f489c-79f3-4244-8067-7e18f1c1b0fe, queue=awx_private_queue)
2021-06-06 22:49:51,619 DEBUG awx.main.dispatch delivered ab8f489c-79f3-4244-8067-7e18f1c1b0fe to worker[183] qsize 0
2021-06-06 22:49:51,622 DEBUG awx.main.dispatch task ab8f489c-79f3-4244-8067-7e18f1c1b0fe starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:49:51,623 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:49:51,638 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:49:51,667 DEBUG awx.main.dispatch task ab8f489c-79f3-4244-8067-7e18f1c1b0fe is finished
RESULT 2
OKREADY
2021-06-06 22:50:11,405 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(37121f49-e4db-4174-b4e9-30f7158e912d, queue=dev-awx-0)
[2021-06-06 22:50:11,405: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(37121f49-e4db-4174-b4e9-30f7158e912d, queue=dev-awx-0)
2021-06-06 22:50:11,633 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c588d754-00df-4365-a02c-9ab50215bb57, queue=dev-awx-0)
[2021-06-06 22:50:11,633: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c588d754-00df-4365-a02c-9ab50215bb57, queue=dev-awx-0)
2021-06-06 22:50:11,646 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(abccc900-ace6-4793-85ca-8e20cc7995b2, queue=awx_private_queue)
[2021-06-06 22:50:11,646: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(abccc900-ace6-4793-85ca-8e20cc7995b2, queue=awx_private_queue)
2021-06-06 22:50:11,659 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a4e5fdd9-ec8f-497a-b228-1b2e544d6e5b, queue=awx_private_queue)
[2021-06-06 22:50:11,659: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a4e5fdd9-ec8f-497a-b228-1b2e544d6e5b, queue=awx_private_queue)
2021-06-06 22:50:11,661 DEBUG awx.main.dispatch delivered 37121f49-e4db-4174-b4e9-30f7158e912d to worker[182] qsize 0
2021-06-06 22:50:11,665 DEBUG awx.main.dispatch delivered c588d754-00df-4365-a02c-9ab50215bb57 to worker[181] qsize 0
2021-06-06 22:50:11,667 DEBUG awx.main.dispatch delivered abccc900-ace6-4793-85ca-8e20cc7995b2 to worker[184] qsize 0
2021-06-06 22:50:11,667 DEBUG awx.main.dispatch task 37121f49-e4db-4174-b4e9-30f7158e912d starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 22:50:11,672 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 22:50:11,669 DEBUG awx.main.dispatch task c588d754-00df-4365-a02c-9ab50215bb57 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 22:50:11,671 DEBUG awx.main.dispatch task abccc900-ace6-4793-85ca-8e20cc7995b2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:50:11,677 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:50:11,679 DEBUG awx.main.dispatch delivered a4e5fdd9-ec8f-497a-b228-1b2e544d6e5b to worker[183] qsize 0
2021-06-06 22:50:11,683 DEBUG awx.main.dispatch task a4e5fdd9-ec8f-497a-b228-1b2e544d6e5b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:50:11,716 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:50:11,725 DEBUG awx.main.dispatch task c588d754-00df-4365-a02c-9ab50215bb57 is finished
2021-06-06 22:50:11,728 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:50:11,725 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:49:41.641089+00:00
2021-06-06 22:50:11,746 DEBUG awx.main.dispatch task 37121f49-e4db-4174-b4e9-30f7158e912d is finished
2021-06-06 22:50:11,780 DEBUG awx.main.dispatch task a4e5fdd9-ec8f-497a-b228-1b2e544d6e5b is finished
2021-06-06 22:50:11,798 DEBUG awx.main.dispatch task abccc900-ace6-4793-85ca-8e20cc7995b2 is finished
2021-06-06 22:50:31,656 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3a81d6c3-fe54-49a0-af01-499ca3cafbe1, queue=awx_private_queue)
[2021-06-06 22:50:31,656: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3a81d6c3-fe54-49a0-af01-499ca3cafbe1, queue=awx_private_queue)
2021-06-06 22:50:31,669 DEBUG awx.main.dispatch delivered 3a81d6c3-fe54-49a0-af01-499ca3cafbe1 to worker[183] qsize 0
2021-06-06 22:50:31,671 DEBUG awx.main.dispatch task 3a81d6c3-fe54-49a0-af01-499ca3cafbe1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:50:31,673 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:50:31,694 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:50:31,732 DEBUG awx.main.dispatch task 3a81d6c3-fe54-49a0-af01-499ca3cafbe1 is finished
2021-06-06 22:50:41,659 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b469d281-dada-4090-831a-7b03fa27fc29, queue=awx_private_queue)
[2021-06-06 22:50:41,659: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b469d281-dada-4090-831a-7b03fa27fc29, queue=awx_private_queue)
2021-06-06 22:50:41,680 DEBUG awx.main.dispatch delivered b469d281-dada-4090-831a-7b03fa27fc29 to worker[184] qsize 0
2021-06-06 22:50:41,682 DEBUG awx.main.dispatch task b469d281-dada-4090-831a-7b03fa27fc29 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:50:41,713 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:50:41,725 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:50:11.716999+00:00
2021-06-06 22:50:41,776 DEBUG awx.main.dispatch task b469d281-dada-4090-831a-7b03fa27fc29 is finished
2021-06-06 22:50:51,656 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2f8d190e-31f2-4959-8e9d-645a1c67923b, queue=awx_private_queue)
[2021-06-06 22:50:51,656: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2f8d190e-31f2-4959-8e9d-645a1c67923b, queue=awx_private_queue)
2021-06-06 22:50:51,670 DEBUG awx.main.dispatch delivered 2f8d190e-31f2-4959-8e9d-645a1c67923b to worker[182] qsize 0
2021-06-06 22:50:51,674 DEBUG awx.main.dispatch task 2f8d190e-31f2-4959-8e9d-645a1c67923b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:50:51,676 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:50:51,702 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:50:51,750 DEBUG awx.main.dispatch task 2f8d190e-31f2-4959-8e9d-645a1c67923b is finished
RESULT 2
OKREADY
2021-06-06 22:51:11,407 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b047b6ee-5bc3-41f2-bf5d-af65965c1796, queue=dev-awx-0)
[2021-06-06 22:51:11,407: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b047b6ee-5bc3-41f2-bf5d-af65965c1796, queue=dev-awx-0)
2021-06-06 22:51:11,436 DEBUG awx.main.dispatch delivered b047b6ee-5bc3-41f2-bf5d-af65965c1796 to worker[181] qsize 0
2021-06-06 22:51:11,438 DEBUG awx.main.dispatch task b047b6ee-5bc3-41f2-bf5d-af65965c1796 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 22:51:11,440 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 22:51:11,508 DEBUG awx.main.dispatch task b047b6ee-5bc3-41f2-bf5d-af65965c1796 is finished
2021-06-06 22:51:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(edf54a0b-42d5-4d68-8add-c2c7a12c53d6, queue=dev-awx-0)
[2021-06-06 22:51:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(edf54a0b-42d5-4d68-8add-c2c7a12c53d6, queue=dev-awx-0)
2021-06-06 22:51:11,655 DEBUG awx.main.dispatch delivered edf54a0b-42d5-4d68-8add-c2c7a12c53d6 to worker[183] qsize 0
2021-06-06 22:51:11,664 DEBUG awx.main.dispatch task edf54a0b-42d5-4d68-8add-c2c7a12c53d6 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 22:51:11,655 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(abe49c30-c0bd-40b7-8a95-6b349094057e, queue=awx_private_queue)
[2021-06-06 22:51:11,655: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(abe49c30-c0bd-40b7-8a95-6b349094057e, queue=awx_private_queue)
2021-06-06 22:51:11,674 DEBUG awx.main.dispatch task edf54a0b-42d5-4d68-8add-c2c7a12c53d6 is finished
2021-06-06 22:51:11,684 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(14dd91e4-04ba-4f92-8c07-f1d9444f68fd, queue=awx_private_queue)
[2021-06-06 22:51:11,684: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(14dd91e4-04ba-4f92-8c07-f1d9444f68fd, queue=awx_private_queue)
2021-06-06 22:51:11,683 DEBUG awx.main.dispatch delivered abe49c30-c0bd-40b7-8a95-6b349094057e to worker[182] qsize 0
2021-06-06 22:51:11,843 DEBUG awx.main.dispatch delivered 14dd91e4-04ba-4f92-8c07-f1d9444f68fd to worker[183] qsize 0
2021-06-06 22:51:11,843 DEBUG awx.main.dispatch task abe49c30-c0bd-40b7-8a95-6b349094057e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:51:11,845 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:51:11,845 DEBUG awx.main.dispatch task 14dd91e4-04ba-4f92-8c07-f1d9444f68fd starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:51:11,853 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:51:11,857 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:50:41.713584+00:00
2021-06-06 22:51:11,857 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:51:11,866 DEBUG awx.main.dispatch task 14dd91e4-04ba-4f92-8c07-f1d9444f68fd is finished
2021-06-06 22:51:11,883 DEBUG awx.main.dispatch task abe49c30-c0bd-40b7-8a95-6b349094057e is finished
2021-06-06 22:51:31,672 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5ebf5522-cd48-4968-acf4-98924f9b2466, queue=awx_private_queue)
[2021-06-06 22:51:31,672: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5ebf5522-cd48-4968-acf4-98924f9b2466, queue=awx_private_queue)
2021-06-06 22:51:31,697 DEBUG awx.main.dispatch delivered 5ebf5522-cd48-4968-acf4-98924f9b2466 to worker[183] qsize 0
2021-06-06 22:51:31,705 DEBUG awx.main.dispatch task 5ebf5522-cd48-4968-acf4-98924f9b2466 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:51:31,708 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:51:31,729 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:51:31,756 DEBUG awx.main.dispatch task 5ebf5522-cd48-4968-acf4-98924f9b2466 is finished
2021-06-06 22:51:41,689 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(04a46d1b-9c74-492c-9278-26cf70e54518, queue=awx_private_queue)
[2021-06-06 22:51:41,689: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(04a46d1b-9c74-492c-9278-26cf70e54518, queue=awx_private_queue)
2021-06-06 22:51:41,713 DEBUG awx.main.dispatch delivered 04a46d1b-9c74-492c-9278-26cf70e54518 to worker[184] qsize 0
2021-06-06 22:51:41,716 DEBUG awx.main.dispatch task 04a46d1b-9c74-492c-9278-26cf70e54518 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:51:41,733 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:51:41,739 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:51:11.853747+00:00
2021-06-06 22:51:41,752 DEBUG awx.main.dispatch task 04a46d1b-9c74-492c-9278-26cf70e54518 is finished
2021-06-06 22:51:51,669 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(28c3d790-bca4-427a-85d5-46553f6869c3, queue=awx_private_queue)
[2021-06-06 22:51:51,669: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(28c3d790-bca4-427a-85d5-46553f6869c3, queue=awx_private_queue)
2021-06-06 22:51:51,690 DEBUG awx.main.dispatch delivered 28c3d790-bca4-427a-85d5-46553f6869c3 to worker[182] qsize 0
2021-06-06 22:51:51,693 DEBUG awx.main.dispatch task 28c3d790-bca4-427a-85d5-46553f6869c3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:51:51,696 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:51:51,712 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:51:51,736 DEBUG awx.main.dispatch task 28c3d790-bca4-427a-85d5-46553f6869c3 is finished
RESULT 2
OKREADY
2021-06-06 22:52:11,412 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(0a1a1435-6dc8-4cfc-bf01-2fbbeaeb3100, queue=dev-awx-0)
[2021-06-06 22:52:11,412: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(0a1a1435-6dc8-4cfc-bf01-2fbbeaeb3100, queue=dev-awx-0)
2021-06-06 22:52:11,453 DEBUG awx.main.dispatch delivered 0a1a1435-6dc8-4cfc-bf01-2fbbeaeb3100 to worker[182] qsize 0
2021-06-06 22:52:11,457 DEBUG awx.main.dispatch task 0a1a1435-6dc8-4cfc-bf01-2fbbeaeb3100 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 22:52:11,461 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 22:52:11,487 DEBUG awx.main.dispatch task 0a1a1435-6dc8-4cfc-bf01-2fbbeaeb3100 is finished
2021-06-06 22:52:11,633 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(7f9db6f1-7bd7-43b7-bd6c-bdddab1b96d7, queue=dev-awx-0)
[2021-06-06 22:52:11,633: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(7f9db6f1-7bd7-43b7-bd6c-bdddab1b96d7, queue=dev-awx-0)
2021-06-06 22:52:11,646 DEBUG awx.main.dispatch delivered 7f9db6f1-7bd7-43b7-bd6c-bdddab1b96d7 to worker[182] qsize 0
2021-06-06 22:52:11,648 DEBUG awx.main.dispatch task 7f9db6f1-7bd7-43b7-bd6c-bdddab1b96d7 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 22:52:11,665 DEBUG awx.main.dispatch task 7f9db6f1-7bd7-43b7-bd6c-bdddab1b96d7 is finished
2021-06-06 22:52:11,668 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4b18c529-4308-410b-85c8-733a3f37b63c, queue=awx_private_queue)
[2021-06-06 22:52:11,668: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4b18c529-4308-410b-85c8-733a3f37b63c, queue=awx_private_queue)
2021-06-06 22:52:11,683 DEBUG awx.main.dispatch delivered 4b18c529-4308-410b-85c8-733a3f37b63c to worker[183] qsize 0
2021-06-06 22:52:11,688 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b1fcba39-f5aa-485a-8190-9a623b337f3d, queue=awx_private_queue)
[2021-06-06 22:52:11,688: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b1fcba39-f5aa-485a-8190-9a623b337f3d, queue=awx_private_queue)
2021-06-06 22:52:11,688 DEBUG awx.main.dispatch task 4b18c529-4308-410b-85c8-733a3f37b63c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:52:11,698 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:52:11,704 DEBUG awx.main.dispatch delivered b1fcba39-f5aa-485a-8190-9a623b337f3d to worker[182] qsize 0
2021-06-06 22:52:11,706 DEBUG awx.main.dispatch task b1fcba39-f5aa-485a-8190-9a623b337f3d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:52:11,714 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:52:11,716 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:51:41.733382+00:00
2021-06-06 22:52:11,721 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:52:11,730 DEBUG awx.main.dispatch task b1fcba39-f5aa-485a-8190-9a623b337f3d is finished
2021-06-06 22:52:11,753 DEBUG awx.main.dispatch task 4b18c529-4308-410b-85c8-733a3f37b63c is finished
2021-06-06 22:52:31,669 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cc1cdc2a-fdf5-4f10-b5c5-bcece5d89d67, queue=awx_private_queue)
[2021-06-06 22:52:31,669: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cc1cdc2a-fdf5-4f10-b5c5-bcece5d89d67, queue=awx_private_queue)
2021-06-06 22:52:31,716 DEBUG awx.main.dispatch delivered cc1cdc2a-fdf5-4f10-b5c5-bcece5d89d67 to worker[184] qsize 0
2021-06-06 22:52:31,720 DEBUG awx.main.dispatch task cc1cdc2a-fdf5-4f10-b5c5-bcece5d89d67 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:52:31,723 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:52:31,742 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:52:31,777 DEBUG awx.main.dispatch task cc1cdc2a-fdf5-4f10-b5c5-bcece5d89d67 is finished
2021-06-06 22:52:41,690 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(258f8206-e126-4fd3-a7bf-fa1c5c926876, queue=awx_private_queue)
[2021-06-06 22:52:41,690: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(258f8206-e126-4fd3-a7bf-fa1c5c926876, queue=awx_private_queue)
2021-06-06 22:52:41,715 DEBUG awx.main.dispatch delivered 258f8206-e126-4fd3-a7bf-fa1c5c926876 to worker[183] qsize 0
2021-06-06 22:52:41,719 DEBUG awx.main.dispatch task 258f8206-e126-4fd3-a7bf-fa1c5c926876 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:52:41,892 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:52:41,894 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:52:11.714278+00:00
2021-06-06 22:52:41,904 DEBUG awx.main.dispatch task 258f8206-e126-4fd3-a7bf-fa1c5c926876 is finished
2021-06-06 22:52:51,673 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b3486f79-1282-4dce-955f-3114e9e00a1a, queue=awx_private_queue)
[2021-06-06 22:52:51,673: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b3486f79-1282-4dce-955f-3114e9e00a1a, queue=awx_private_queue)
2021-06-06 22:52:51,698 DEBUG awx.main.dispatch delivered b3486f79-1282-4dce-955f-3114e9e00a1a to worker[182] qsize 0
2021-06-06 22:52:51,704 DEBUG awx.main.dispatch task b3486f79-1282-4dce-955f-3114e9e00a1a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:52:51,708 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:52:51,738 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:52:51,770 DEBUG awx.main.dispatch task b3486f79-1282-4dce-955f-3114e9e00a1a is finished
RESULT 2
OKREADY
2021-06-06 22:53:11,416 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(dbaf0aff-9cd5-4124-9f06-f22bde43735d, queue=dev-awx-0)
[2021-06-06 22:53:11,416: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(dbaf0aff-9cd5-4124-9f06-f22bde43735d, queue=dev-awx-0)
2021-06-06 22:53:11,448 DEBUG awx.main.dispatch delivered dbaf0aff-9cd5-4124-9f06-f22bde43735d to worker[183] qsize 0
2021-06-06 22:53:11,451 DEBUG awx.main.dispatch task dbaf0aff-9cd5-4124-9f06-f22bde43735d starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 22:53:11,453 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 22:53:11,482 DEBUG awx.main.dispatch task dbaf0aff-9cd5-4124-9f06-f22bde43735d is finished
2021-06-06 22:53:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c0f1e760-64c6-4eed-98e5-d79d8c6229df, queue=dev-awx-0)
[2021-06-06 22:53:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c0f1e760-64c6-4eed-98e5-d79d8c6229df, queue=dev-awx-0)
2021-06-06 22:53:11,646 DEBUG awx.main.dispatch delivered c0f1e760-64c6-4eed-98e5-d79d8c6229df to worker[184] qsize 0
2021-06-06 22:53:11,649 DEBUG awx.main.dispatch task c0f1e760-64c6-4eed-98e5-d79d8c6229df starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 22:53:11,666 DEBUG awx.main.dispatch task c0f1e760-64c6-4eed-98e5-d79d8c6229df is finished
2021-06-06 22:53:11,669 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b8f06851-64d0-4281-98cc-028619c74400, queue=awx_private_queue)
[2021-06-06 22:53:11,669: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b8f06851-64d0-4281-98cc-028619c74400, queue=awx_private_queue)
2021-06-06 22:53:11,682 DEBUG awx.main.dispatch delivered b8f06851-64d0-4281-98cc-028619c74400 to worker[183] qsize 0
2021-06-06 22:53:11,684 DEBUG awx.main.dispatch task b8f06851-64d0-4281-98cc-028619c74400 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:53:11,685 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:53:11,689 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d12d81d8-9660-406b-9ad5-9679a1abfab9, queue=awx_private_queue)
[2021-06-06 22:53:11,689: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d12d81d8-9660-406b-9ad5-9679a1abfab9, queue=awx_private_queue)
2021-06-06 22:53:11,699 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:53:11,709 DEBUG awx.main.dispatch delivered d12d81d8-9660-406b-9ad5-9679a1abfab9 to worker[181] qsize 0
2021-06-06 22:53:11,713 DEBUG awx.main.dispatch task d12d81d8-9660-406b-9ad5-9679a1abfab9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:53:11,725 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:53:11,728 DEBUG awx.main.dispatch task b8f06851-64d0-4281-98cc-028619c74400 is finished
2021-06-06 22:53:11,730 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:52:41.892552+00:00
2021-06-06 22:53:11,750 DEBUG awx.main.dispatch task d12d81d8-9660-406b-9ad5-9679a1abfab9 is finished
2021-06-06 22:53:31,685 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3ac9b38f-6184-4606-9001-05c55ee024e3, queue=awx_private_queue)
[2021-06-06 22:53:31,685: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3ac9b38f-6184-4606-9001-05c55ee024e3, queue=awx_private_queue)
2021-06-06 22:53:31,707 DEBUG awx.main.dispatch delivered 3ac9b38f-6184-4606-9001-05c55ee024e3 to worker[181] qsize 0
2021-06-06 22:53:31,713 DEBUG awx.main.dispatch task 3ac9b38f-6184-4606-9001-05c55ee024e3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:53:31,716 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:53:31,737 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:53:31,763 DEBUG awx.main.dispatch task 3ac9b38f-6184-4606-9001-05c55ee024e3 is finished
2021-06-06 22:53:41,690 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(033d995e-8f35-494c-8ead-6c02b0285186, queue=awx_private_queue)
[2021-06-06 22:53:41,690: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(033d995e-8f35-494c-8ead-6c02b0285186, queue=awx_private_queue)
2021-06-06 22:53:41,726 DEBUG awx.main.dispatch delivered 033d995e-8f35-494c-8ead-6c02b0285186 to worker[182] qsize 0
2021-06-06 22:53:41,729 DEBUG awx.main.dispatch task 033d995e-8f35-494c-8ead-6c02b0285186 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:53:41,739 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:53:41,743 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:53:11.725629+00:00
2021-06-06 22:53:41,755 DEBUG awx.main.dispatch task 033d995e-8f35-494c-8ead-6c02b0285186 is finished
2021-06-06 22:53:51,681 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f2829c24-0427-4180-b09d-986f68e25d10, queue=awx_private_queue)
[2021-06-06 22:53:51,681: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f2829c24-0427-4180-b09d-986f68e25d10, queue=awx_private_queue)
2021-06-06 22:53:51,695 DEBUG awx.main.dispatch delivered f2829c24-0427-4180-b09d-986f68e25d10 to worker[181] qsize 0
2021-06-06 22:53:51,697 DEBUG awx.main.dispatch task f2829c24-0427-4180-b09d-986f68e25d10 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:53:51,699 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:53:51,713 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:53:51,743 DEBUG awx.main.dispatch task f2829c24-0427-4180-b09d-986f68e25d10 is finished
RESULT 2
OKREADY
2021-06-06 22:54:11,428 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(78122d04-556a-4bb9-8e9b-c60984bc61a3, queue=dev-awx-0)
[2021-06-06 22:54:11,428: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(78122d04-556a-4bb9-8e9b-c60984bc61a3, queue=dev-awx-0)
2021-06-06 22:54:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5c9609b3-e80d-4b71-9d4a-5c27f2c0c7e2, queue=dev-awx-0)
[2021-06-06 22:54:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5c9609b3-e80d-4b71-9d4a-5c27f2c0c7e2, queue=dev-awx-0)
2021-06-06 22:54:11,644 DEBUG awx.main.dispatch delivered 78122d04-556a-4bb9-8e9b-c60984bc61a3 to worker[181] qsize 0
2021-06-06 22:54:11,648 DEBUG awx.main.dispatch delivered 5c9609b3-e80d-4b71-9d4a-5c27f2c0c7e2 to worker[182] qsize 0
2021-06-06 22:54:11,650 DEBUG awx.main.dispatch task 78122d04-556a-4bb9-8e9b-c60984bc61a3 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 22:54:11,653 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 22:54:11,651 DEBUG awx.main.dispatch task 5c9609b3-e80d-4b71-9d4a-5c27f2c0c7e2 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 22:54:11,668 DEBUG awx.main.dispatch task 5c9609b3-e80d-4b71-9d4a-5c27f2c0c7e2 is finished
2021-06-06 22:54:11,681 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c4bd20e5-b396-495f-8069-5483cf476ae7, queue=awx_private_queue)
[2021-06-06 22:54:11,681: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c4bd20e5-b396-495f-8069-5483cf476ae7, queue=awx_private_queue)
2021-06-06 22:54:11,693 DEBUG awx.main.dispatch delivered c4bd20e5-b396-495f-8069-5483cf476ae7 to worker[184] qsize 0
2021-06-06 22:54:11,694 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6f3badc5-36e3-412a-9bec-de27275a7643, queue=awx_private_queue)
[2021-06-06 22:54:11,694: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6f3badc5-36e3-412a-9bec-de27275a7643, queue=awx_private_queue)
2021-06-06 22:54:11,696 DEBUG awx.main.dispatch task c4bd20e5-b396-495f-8069-5483cf476ae7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:54:11,700 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:54:11,707 DEBUG awx.main.dispatch delivered 6f3badc5-36e3-412a-9bec-de27275a7643 to worker[183] qsize 0
2021-06-06 22:54:11,711 DEBUG awx.main.dispatch task 6f3badc5-36e3-412a-9bec-de27275a7643 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:54:11,720 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:54:11,725 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:54:11,729 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:53:41.739390+00:00
2021-06-06 22:54:11,744 DEBUG awx.main.dispatch task c4bd20e5-b396-495f-8069-5483cf476ae7 is finished
2021-06-06 22:54:11,764 DEBUG awx.main.dispatch task 78122d04-556a-4bb9-8e9b-c60984bc61a3 is finished
2021-06-06 22:54:11,776 DEBUG awx.main.dispatch task 6f3badc5-36e3-412a-9bec-de27275a7643 is finished
2021-06-06 22:54:31,693 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0f05243a-8c5e-4125-8d59-98ed4971318d, queue=awx_private_queue)
[2021-06-06 22:54:31,693: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0f05243a-8c5e-4125-8d59-98ed4971318d, queue=awx_private_queue)
2021-06-06 22:54:31,711 DEBUG awx.main.dispatch delivered 0f05243a-8c5e-4125-8d59-98ed4971318d to worker[181] qsize 0
2021-06-06 22:54:31,714 DEBUG awx.main.dispatch task 0f05243a-8c5e-4125-8d59-98ed4971318d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:54:31,716 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:54:31,737 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:54:31,763 DEBUG awx.main.dispatch task 0f05243a-8c5e-4125-8d59-98ed4971318d is finished
2021-06-06 22:54:41,694 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f5ca8f99-16c0-4aab-a4b3-1321a8ef3c35, queue=awx_private_queue)
[2021-06-06 22:54:41,694: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f5ca8f99-16c0-4aab-a4b3-1321a8ef3c35, queue=awx_private_queue)
2021-06-06 22:54:41,708 DEBUG awx.main.dispatch delivered f5ca8f99-16c0-4aab-a4b3-1321a8ef3c35 to worker[181] qsize 0
2021-06-06 22:54:41,710 DEBUG awx.main.dispatch task f5ca8f99-16c0-4aab-a4b3-1321a8ef3c35 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:54:41,719 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:54:41,722 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:54:11.725572+00:00
2021-06-06 22:54:41,734 DEBUG awx.main.dispatch task f5ca8f99-16c0-4aab-a4b3-1321a8ef3c35 is finished
2021-06-06 22:54:51,693 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9d9256d1-b06c-4dd5-9012-a9de480a8fd6, queue=awx_private_queue)
[2021-06-06 22:54:51,693: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9d9256d1-b06c-4dd5-9012-a9de480a8fd6, queue=awx_private_queue)
2021-06-06 22:54:51,705 DEBUG awx.main.dispatch delivered 9d9256d1-b06c-4dd5-9012-a9de480a8fd6 to worker[183] qsize 0
2021-06-06 22:54:51,707 DEBUG awx.main.dispatch task 9d9256d1-b06c-4dd5-9012-a9de480a8fd6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:54:51,709 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:54:51,728 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:54:51,761 DEBUG awx.main.dispatch task 9d9256d1-b06c-4dd5-9012-a9de480a8fd6 is finished
RESULT 2
OKREADY
2021-06-06 22:55:11,440 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b2ac7bce-f2a4-4941-bc0a-56371fbb5cfc, queue=dev-awx-0)
[2021-06-06 22:55:11,440: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b2ac7bce-f2a4-4941-bc0a-56371fbb5cfc, queue=dev-awx-0)
2021-06-06 22:55:11,493 DEBUG awx.main.dispatch delivered b2ac7bce-f2a4-4941-bc0a-56371fbb5cfc to worker[183] qsize 0
2021-06-06 22:55:11,498 DEBUG awx.main.dispatch task b2ac7bce-f2a4-4941-bc0a-56371fbb5cfc starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 22:55:11,501 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 22:55:11,526 DEBUG awx.main.dispatch task b2ac7bce-f2a4-4941-bc0a-56371fbb5cfc is finished
2021-06-06 22:55:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(868be09b-6190-4d3e-863b-09029954e105, queue=dev-awx-0)
[2021-06-06 22:55:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(868be09b-6190-4d3e-863b-09029954e105, queue=dev-awx-0)
2021-06-06 22:55:11,679 DEBUG awx.main.dispatch delivered 868be09b-6190-4d3e-863b-09029954e105 to worker[182] qsize 0
2021-06-06 22:55:11,693 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(439b2559-d576-4bed-beba-f3709d9a0b5e, queue=awx_private_queue)
[2021-06-06 22:55:11,693: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(439b2559-d576-4bed-beba-f3709d9a0b5e, queue=awx_private_queue)
2021-06-06 22:55:11,707 DEBUG awx.main.dispatch delivered 439b2559-d576-4bed-beba-f3709d9a0b5e to worker[184] qsize 0
2021-06-06 22:55:11,708 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(47ea9676-27a1-4b3e-9840-4d5698a9df1f, queue=awx_private_queue)
[2021-06-06 22:55:11,708: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(47ea9676-27a1-4b3e-9840-4d5698a9df1f, queue=awx_private_queue)
2021-06-06 22:55:11,721 DEBUG awx.main.dispatch delivered 47ea9676-27a1-4b3e-9840-4d5698a9df1f to worker[181] qsize 0
2021-06-06 22:55:11,725 DEBUG awx.main.dispatch task 47ea9676-27a1-4b3e-9840-4d5698a9df1f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:55:11,737 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:55:11,742 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:54:41.720332+00:00
2021-06-06 22:55:11,759 DEBUG awx.main.dispatch task 47ea9676-27a1-4b3e-9840-4d5698a9df1f is finished
2021-06-06 22:55:11,683 DEBUG awx.main.dispatch task 868be09b-6190-4d3e-863b-09029954e105 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 22:55:11,710 DEBUG awx.main.dispatch task 439b2559-d576-4bed-beba-f3709d9a0b5e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:55:11,906 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:55:11,909 DEBUG awx.main.dispatch task 868be09b-6190-4d3e-863b-09029954e105 is finished
2021-06-06 22:55:11,915 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:55:11,943 DEBUG awx.main.dispatch task 439b2559-d576-4bed-beba-f3709d9a0b5e is finished
2021-06-06 22:55:31,694 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ea00f813-81f7-4fae-952f-b58291402f35, queue=awx_private_queue)
[2021-06-06 22:55:31,694: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ea00f813-81f7-4fae-952f-b58291402f35, queue=awx_private_queue)
2021-06-06 22:55:31,713 DEBUG awx.main.dispatch delivered ea00f813-81f7-4fae-952f-b58291402f35 to worker[181] qsize 0
2021-06-06 22:55:31,717 DEBUG awx.main.dispatch task ea00f813-81f7-4fae-952f-b58291402f35 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:55:31,719 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:55:31,736 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:55:31,761 DEBUG awx.main.dispatch task ea00f813-81f7-4fae-952f-b58291402f35 is finished
2021-06-06 22:55:41,709 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9f5766ae-7902-4da5-9434-c433b6e5871a, queue=awx_private_queue)
[2021-06-06 22:55:41,709: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9f5766ae-7902-4da5-9434-c433b6e5871a, queue=awx_private_queue)
2021-06-06 22:55:41,726 DEBUG awx.main.dispatch delivered 9f5766ae-7902-4da5-9434-c433b6e5871a to worker[184] qsize 0
2021-06-06 22:55:41,729 DEBUG awx.main.dispatch task 9f5766ae-7902-4da5-9434-c433b6e5871a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:55:41,738 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:55:41,743 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:55:11.737677+00:00
2021-06-06 22:55:41,760 DEBUG awx.main.dispatch task 9f5766ae-7902-4da5-9434-c433b6e5871a is finished
2021-06-06 22:55:51,693 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0edd6284-1f6b-44e5-9067-f373d7029433, queue=awx_private_queue)
[2021-06-06 22:55:51,693: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0edd6284-1f6b-44e5-9067-f373d7029433, queue=awx_private_queue)
2021-06-06 22:55:51,709 DEBUG awx.main.dispatch delivered 0edd6284-1f6b-44e5-9067-f373d7029433 to worker[181] qsize 0
2021-06-06 22:55:51,712 DEBUG awx.main.dispatch task 0edd6284-1f6b-44e5-9067-f373d7029433 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:55:51,713 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:55:51,731 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:55:51,769 DEBUG awx.main.dispatch task 0edd6284-1f6b-44e5-9067-f373d7029433 is finished
RESULT 2
OKREADY
2021-06-06 22:56:11,443 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c1f2fbdc-62cf-426f-8b02-a48d807d3900, queue=dev-awx-0)
[2021-06-06 22:56:11,443: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c1f2fbdc-62cf-426f-8b02-a48d807d3900, queue=dev-awx-0)
2021-06-06 22:56:11,466 DEBUG awx.main.dispatch delivered c1f2fbdc-62cf-426f-8b02-a48d807d3900 to worker[183] qsize 0
2021-06-06 22:56:11,468 DEBUG awx.main.dispatch task c1f2fbdc-62cf-426f-8b02-a48d807d3900 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 22:56:11,471 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 22:56:11,487 DEBUG awx.main.dispatch task c1f2fbdc-62cf-426f-8b02-a48d807d3900 is finished
2021-06-06 22:56:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(561ba39b-d7d9-4b01-9726-98d8ad55b3da, queue=dev-awx-0)
[2021-06-06 22:56:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(561ba39b-d7d9-4b01-9726-98d8ad55b3da, queue=dev-awx-0)
2021-06-06 22:56:11,649 DEBUG awx.main.dispatch delivered 561ba39b-d7d9-4b01-9726-98d8ad55b3da to worker[182] qsize 0
2021-06-06 22:56:11,651 DEBUG awx.main.dispatch task 561ba39b-d7d9-4b01-9726-98d8ad55b3da starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 22:56:11,660 DEBUG awx.main.dispatch task 561ba39b-d7d9-4b01-9726-98d8ad55b3da is finished
2021-06-06 22:56:11,693 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(479041bf-0a17-4716-ae5c-d159c12db1f2, queue=awx_private_queue)
[2021-06-06 22:56:11,693: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(479041bf-0a17-4716-ae5c-d159c12db1f2, queue=awx_private_queue)
2021-06-06 22:56:11,705 DEBUG awx.main.dispatch delivered 479041bf-0a17-4716-ae5c-d159c12db1f2 to worker[181] qsize 0
2021-06-06 22:56:11,707 DEBUG awx.main.dispatch task 479041bf-0a17-4716-ae5c-d159c12db1f2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:56:11,708 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:56:11,708 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e51e703d-df37-4965-bfb3-441b7a828891, queue=awx_private_queue)
[2021-06-06 22:56:11,708: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e51e703d-df37-4965-bfb3-441b7a828891, queue=awx_private_queue)
2021-06-06 22:56:11,720 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:56:11,725 DEBUG awx.main.dispatch delivered e51e703d-df37-4965-bfb3-441b7a828891 to worker[183] qsize 0
2021-06-06 22:56:11,726 DEBUG awx.main.dispatch task e51e703d-df37-4965-bfb3-441b7a828891 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:56:11,736 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:56:11,738 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:55:41.738503+00:00
2021-06-06 22:56:11,743 DEBUG awx.main.dispatch task 479041bf-0a17-4716-ae5c-d159c12db1f2 is finished
2021-06-06 22:56:11,751 DEBUG awx.main.dispatch task e51e703d-df37-4965-bfb3-441b7a828891 is finished
2021-06-06 22:56:31,703 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(79695f8e-37c3-4eca-b17b-4fcfa6e12fcb, queue=awx_private_queue)
[2021-06-06 22:56:31,703: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(79695f8e-37c3-4eca-b17b-4fcfa6e12fcb, queue=awx_private_queue)
2021-06-06 22:56:31,726 DEBUG awx.main.dispatch delivered 79695f8e-37c3-4eca-b17b-4fcfa6e12fcb to worker[183] qsize 0
2021-06-06 22:56:31,895 DEBUG awx.main.dispatch task 79695f8e-37c3-4eca-b17b-4fcfa6e12fcb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:56:31,897 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:56:31,915 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:56:31,956 DEBUG awx.main.dispatch task 79695f8e-37c3-4eca-b17b-4fcfa6e12fcb is finished
2021-06-06 22:56:41,714 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d4b5d9b7-15aa-4390-892e-0b8265dd1d0d, queue=awx_private_queue)
[2021-06-06 22:56:41,714: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d4b5d9b7-15aa-4390-892e-0b8265dd1d0d, queue=awx_private_queue)
2021-06-06 22:56:41,736 DEBUG awx.main.dispatch delivered d4b5d9b7-15aa-4390-892e-0b8265dd1d0d to worker[181] qsize 0
2021-06-06 22:56:41,742 DEBUG awx.main.dispatch task d4b5d9b7-15aa-4390-892e-0b8265dd1d0d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:56:41,757 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:56:41,761 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:56:11.736420+00:00
2021-06-06 22:56:41,776 DEBUG awx.main.dispatch task d4b5d9b7-15aa-4390-892e-0b8265dd1d0d is finished
2021-06-06 22:56:51,704 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(016a2d42-72ee-4f46-961a-1cbf13fe8a19, queue=awx_private_queue)
[2021-06-06 22:56:51,704: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(016a2d42-72ee-4f46-961a-1cbf13fe8a19, queue=awx_private_queue)
2021-06-06 22:56:51,717 DEBUG awx.main.dispatch delivered 016a2d42-72ee-4f46-961a-1cbf13fe8a19 to worker[183] qsize 0
2021-06-06 22:56:51,721 DEBUG awx.main.dispatch task 016a2d42-72ee-4f46-961a-1cbf13fe8a19 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:56:51,724 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:56:51,741 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:56:51,768 DEBUG awx.main.dispatch task 016a2d42-72ee-4f46-961a-1cbf13fe8a19 is finished
RESULT 2
OKREADY
2021-06-06 22:57:11,454 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f0db1dc4-ef2c-480e-920e-0ab38041420e, queue=dev-awx-0)
[2021-06-06 22:57:11,454: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f0db1dc4-ef2c-480e-920e-0ab38041420e, queue=dev-awx-0)
2021-06-06 22:57:11,485 DEBUG awx.main.dispatch delivered f0db1dc4-ef2c-480e-920e-0ab38041420e to worker[181] qsize 0
2021-06-06 22:57:11,487 DEBUG awx.main.dispatch task f0db1dc4-ef2c-480e-920e-0ab38041420e starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 22:57:11,488 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 22:57:11,508 DEBUG awx.main.dispatch task f0db1dc4-ef2c-480e-920e-0ab38041420e is finished
2021-06-06 22:57:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d56b02b9-8e82-4315-b5fe-28918ce89188, queue=dev-awx-0)
[2021-06-06 22:57:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d56b02b9-8e82-4315-b5fe-28918ce89188, queue=dev-awx-0)
2021-06-06 22:57:11,651 DEBUG awx.main.dispatch delivered d56b02b9-8e82-4315-b5fe-28918ce89188 to worker[184] qsize 0
2021-06-06 22:57:11,655 DEBUG awx.main.dispatch task d56b02b9-8e82-4315-b5fe-28918ce89188 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 22:57:11,671 DEBUG awx.main.dispatch task d56b02b9-8e82-4315-b5fe-28918ce89188 is finished
2021-06-06 22:57:11,704 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(64c7833d-950a-4762-9d8b-a36e9a94ec50, queue=awx_private_queue)
[2021-06-06 22:57:11,704: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(64c7833d-950a-4762-9d8b-a36e9a94ec50, queue=awx_private_queue)
2021-06-06 22:57:11,716 DEBUG awx.main.dispatch delivered 64c7833d-950a-4762-9d8b-a36e9a94ec50 to worker[184] qsize 0
2021-06-06 22:57:11,716 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(83d310b1-c24b-4eb0-a879-a333dc9ed237, queue=awx_private_queue)
[2021-06-06 22:57:11,716: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(83d310b1-c24b-4eb0-a879-a333dc9ed237, queue=awx_private_queue)
2021-06-06 22:57:11,717 DEBUG awx.main.dispatch task 64c7833d-950a-4762-9d8b-a36e9a94ec50 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:57:11,719 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:57:11,729 DEBUG awx.main.dispatch delivered 83d310b1-c24b-4eb0-a879-a333dc9ed237 to worker[182] qsize 0
2021-06-06 22:57:11,732 DEBUG awx.main.dispatch task 83d310b1-c24b-4eb0-a879-a333dc9ed237 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:57:11,735 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:57:11,744 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:57:11,747 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:56:41.757221+00:00
2021-06-06 22:57:11,759 DEBUG awx.main.dispatch task 83d310b1-c24b-4eb0-a879-a333dc9ed237 is finished
2021-06-06 22:57:11,760 DEBUG awx.main.dispatch task 64c7833d-950a-4762-9d8b-a36e9a94ec50 is finished
2021-06-06 22:57:31,717 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5e867662-539d-4512-baf3-a89187b76fab, queue=awx_private_queue)
[2021-06-06 22:57:31,717: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5e867662-539d-4512-baf3-a89187b76fab, queue=awx_private_queue)
2021-06-06 22:57:31,759 DEBUG awx.main.dispatch delivered 5e867662-539d-4512-baf3-a89187b76fab to worker[184] qsize 0
2021-06-06 22:57:31,927 DEBUG awx.main.dispatch task 5e867662-539d-4512-baf3-a89187b76fab starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:57:31,928 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:57:31,943 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:57:31,973 DEBUG awx.main.dispatch task 5e867662-539d-4512-baf3-a89187b76fab is finished
2021-06-06 22:57:41,718 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4e865a13-7049-40ee-abc6-a7331ef68b8f, queue=awx_private_queue)
[2021-06-06 22:57:41,718: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4e865a13-7049-40ee-abc6-a7331ef68b8f, queue=awx_private_queue)
2021-06-06 22:57:41,738 DEBUG awx.main.dispatch delivered 4e865a13-7049-40ee-abc6-a7331ef68b8f to worker[182] qsize 0
2021-06-06 22:57:41,740 DEBUG awx.main.dispatch task 4e865a13-7049-40ee-abc6-a7331ef68b8f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:57:41,747 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:57:41,750 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:57:11.744199+00:00
2021-06-06 22:57:41,761 DEBUG awx.main.dispatch task 4e865a13-7049-40ee-abc6-a7331ef68b8f is finished
2021-06-06 22:57:51,718 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fcd1cf22-fdd8-43da-843d-42b1350562e2, queue=awx_private_queue)
[2021-06-06 22:57:51,718: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fcd1cf22-fdd8-43da-843d-42b1350562e2, queue=awx_private_queue)
2021-06-06 22:57:51,739 DEBUG awx.main.dispatch delivered fcd1cf22-fdd8-43da-843d-42b1350562e2 to worker[183] qsize 0
2021-06-06 22:57:51,741 DEBUG awx.main.dispatch task fcd1cf22-fdd8-43da-843d-42b1350562e2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:57:51,743 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:57:51,760 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:57:51,788 DEBUG awx.main.dispatch task fcd1cf22-fdd8-43da-843d-42b1350562e2 is finished
RESULT 2
OKREADY
2021-06-06 22:58:11,393 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(81a9880d-4bb7-4d20-8fa6-4b8fec775b90, queue=dev-awx-0)
[2021-06-06 22:58:11,393: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(81a9880d-4bb7-4d20-8fa6-4b8fec775b90, queue=dev-awx-0)
2021-06-06 22:58:11,411 DEBUG awx.main.dispatch delivered 81a9880d-4bb7-4d20-8fa6-4b8fec775b90 to worker[183] qsize 0
2021-06-06 22:58:11,417 DEBUG awx.main.dispatch task 81a9880d-4bb7-4d20-8fa6-4b8fec775b90 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-06-06 22:58:11,419 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-06-06 22:58:11,441 DEBUG awx.main.dispatch task 81a9880d-4bb7-4d20-8fa6-4b8fec775b90 is finished
2021-06-06 22:58:11,453 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(87b710de-4558-4bf9-8550-3b7dba7432c0, queue=dev-awx-0)
[2021-06-06 22:58:11,453: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(87b710de-4558-4bf9-8550-3b7dba7432c0, queue=dev-awx-0)
2021-06-06 22:58:11,485 DEBUG awx.main.dispatch delivered 87b710de-4558-4bf9-8550-3b7dba7432c0 to worker[181] qsize 0
2021-06-06 22:58:11,488 DEBUG awx.main.dispatch task 87b710de-4558-4bf9-8550-3b7dba7432c0 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 22:58:11,491 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 22:58:11,512 DEBUG awx.main.dispatch task 87b710de-4558-4bf9-8550-3b7dba7432c0 is finished
2021-06-06 22:58:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(4cfd57b8-9748-44a0-9e6d-ef9882e32d72, queue=dev-awx-0)
[2021-06-06 22:58:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(4cfd57b8-9748-44a0-9e6d-ef9882e32d72, queue=dev-awx-0)
2021-06-06 22:58:11,647 DEBUG awx.main.dispatch delivered 4cfd57b8-9748-44a0-9e6d-ef9882e32d72 to worker[181] qsize 0
2021-06-06 22:58:11,649 DEBUG awx.main.dispatch task 4cfd57b8-9748-44a0-9e6d-ef9882e32d72 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 22:58:11,659 DEBUG awx.main.dispatch task 4cfd57b8-9748-44a0-9e6d-ef9882e32d72 is finished
2021-06-06 22:58:11,717 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a6723517-bc56-4262-8934-2dbdc45eda06, queue=awx_private_queue)
[2021-06-06 22:58:11,717: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a6723517-bc56-4262-8934-2dbdc45eda06, queue=awx_private_queue)
2021-06-06 22:58:11,730 DEBUG awx.main.dispatch delivered a6723517-bc56-4262-8934-2dbdc45eda06 to worker[182] qsize 0
2021-06-06 22:58:11,731 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4eb0ba0b-9c19-43c8-9f09-e8ae4e42f1bd, queue=awx_private_queue)
[2021-06-06 22:58:11,731: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4eb0ba0b-9c19-43c8-9f09-e8ae4e42f1bd, queue=awx_private_queue)
2021-06-06 22:58:11,734 DEBUG awx.main.dispatch task a6723517-bc56-4262-8934-2dbdc45eda06 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:58:11,743 DEBUG awx.main.dispatch delivered 4eb0ba0b-9c19-43c8-9f09-e8ae4e42f1bd to worker[181] qsize 0
2021-06-06 22:58:11,744 DEBUG awx.main.dispatch task 4eb0ba0b-9c19-43c8-9f09-e8ae4e42f1bd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:58:11,745 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:58:11,750 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:58:11,752 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:57:41.748031+00:00
2021-06-06 22:58:11,762 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:58:11,763 DEBUG awx.main.dispatch task a6723517-bc56-4262-8934-2dbdc45eda06 is finished
2021-06-06 22:58:11,799 DEBUG awx.main.dispatch task 4eb0ba0b-9c19-43c8-9f09-e8ae4e42f1bd is finished
2021-06-06 22:58:31,738 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a60d3755-49a4-4a2c-9099-020c5d0b6bce, queue=awx_private_queue)
[2021-06-06 22:58:31,738: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a60d3755-49a4-4a2c-9099-020c5d0b6bce, queue=awx_private_queue)
2021-06-06 22:58:31,798 DEBUG awx.main.dispatch delivered a60d3755-49a4-4a2c-9099-020c5d0b6bce to worker[181] qsize 0
2021-06-06 22:58:31,981 DEBUG awx.main.dispatch task a60d3755-49a4-4a2c-9099-020c5d0b6bce starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:58:31,983 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:58:31,999 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:58:32,026 DEBUG awx.main.dispatch task a60d3755-49a4-4a2c-9099-020c5d0b6bce is finished
2021-06-06 22:58:41,718 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(81249f4b-5429-4f4d-8b0a-d2c70dc9345c, queue=awx_private_queue)
[2021-06-06 22:58:41,718: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(81249f4b-5429-4f4d-8b0a-d2c70dc9345c, queue=awx_private_queue)
2021-06-06 22:58:41,730 DEBUG awx.main.dispatch delivered 81249f4b-5429-4f4d-8b0a-d2c70dc9345c to worker[184] qsize 0
2021-06-06 22:58:41,734 DEBUG awx.main.dispatch task 81249f4b-5429-4f4d-8b0a-d2c70dc9345c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:58:41,747 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:58:41,751 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:58:11.750177+00:00
2021-06-06 22:58:41,771 DEBUG awx.main.dispatch task 81249f4b-5429-4f4d-8b0a-d2c70dc9345c is finished
2021-06-06 22:58:51,733 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ac0a326d-f1d4-48ac-977d-c14157682b8e, queue=awx_private_queue)
[2021-06-06 22:58:51,733: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ac0a326d-f1d4-48ac-977d-c14157682b8e, queue=awx_private_queue)
2021-06-06 22:58:51,746 DEBUG awx.main.dispatch delivered ac0a326d-f1d4-48ac-977d-c14157682b8e to worker[181] qsize 0
2021-06-06 22:58:51,748 DEBUG awx.main.dispatch task ac0a326d-f1d4-48ac-977d-c14157682b8e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:58:51,749 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:58:51,763 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:58:51,788 DEBUG awx.main.dispatch task ac0a326d-f1d4-48ac-977d-c14157682b8e is finished
RESULT 2
OKREADY
2021-06-06 22:59:11,470 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(dcb339a3-12f9-45d0-aff7-5e38d4691e83, queue=dev-awx-0)
[2021-06-06 22:59:11,470: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(dcb339a3-12f9-45d0-aff7-5e38d4691e83, queue=dev-awx-0)
2021-06-06 22:59:11,517 DEBUG awx.main.dispatch delivered dcb339a3-12f9-45d0-aff7-5e38d4691e83 to worker[183] qsize 0
2021-06-06 22:59:11,522 DEBUG awx.main.dispatch task dcb339a3-12f9-45d0-aff7-5e38d4691e83 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 22:59:11,527 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 22:59:11,625 DEBUG awx.main.dispatch task dcb339a3-12f9-45d0-aff7-5e38d4691e83 is finished
2021-06-06 22:59:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(75b72a21-3180-41d7-83b0-aadaf60cb243, queue=dev-awx-0)
[2021-06-06 22:59:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(75b72a21-3180-41d7-83b0-aadaf60cb243, queue=dev-awx-0)
2021-06-06 22:59:11,648 DEBUG awx.main.dispatch delivered 75b72a21-3180-41d7-83b0-aadaf60cb243 to worker[182] qsize 0
2021-06-06 22:59:11,651 DEBUG awx.main.dispatch task 75b72a21-3180-41d7-83b0-aadaf60cb243 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 22:59:11,718 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b7c96381-752a-4360-a257-3c2ba115839c, queue=awx_private_queue)
[2021-06-06 22:59:11,718: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b7c96381-752a-4360-a257-3c2ba115839c, queue=awx_private_queue)
2021-06-06 22:59:11,731 DEBUG awx.main.dispatch delivered b7c96381-752a-4360-a257-3c2ba115839c to worker[184] qsize 0
2021-06-06 22:59:11,732 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(693b6521-1bbd-46bb-b615-7795a6a1a636, queue=awx_private_queue)
[2021-06-06 22:59:11,732: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(693b6521-1bbd-46bb-b615-7795a6a1a636, queue=awx_private_queue)
2021-06-06 22:59:11,734 DEBUG awx.main.dispatch task b7c96381-752a-4360-a257-3c2ba115839c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:59:11,745 DEBUG awx.main.dispatch delivered 693b6521-1bbd-46bb-b615-7795a6a1a636 to worker[183] qsize 0
2021-06-06 22:59:11,747 DEBUG awx.main.dispatch task 693b6521-1bbd-46bb-b615-7795a6a1a636 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:59:11,748 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:59:11,780 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:59:11,785 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:58:41.747379+00:00
2021-06-06 22:59:11,789 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:59:11,789 DEBUG awx.main.dispatch task 75b72a21-3180-41d7-83b0-aadaf60cb243 is finished
2021-06-06 22:59:11,836 DEBUG awx.main.dispatch task b7c96381-752a-4360-a257-3c2ba115839c is finished
2021-06-06 22:59:11,844 DEBUG awx.main.dispatch task 693b6521-1bbd-46bb-b615-7795a6a1a636 is finished
2021-06-06 22:59:31,733 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(582b55c6-5d90-4bda-a4b9-7c36a80a0c9f, queue=awx_private_queue)
[2021-06-06 22:59:31,733: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(582b55c6-5d90-4bda-a4b9-7c36a80a0c9f, queue=awx_private_queue)
2021-06-06 22:59:31,767 DEBUG awx.main.dispatch delivered 582b55c6-5d90-4bda-a4b9-7c36a80a0c9f to worker[181] qsize 0
2021-06-06 22:59:31,770 DEBUG awx.main.dispatch task 582b55c6-5d90-4bda-a4b9-7c36a80a0c9f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:59:31,938 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:59:31,947 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:59:31,978 DEBUG awx.main.dispatch task 582b55c6-5d90-4bda-a4b9-7c36a80a0c9f is finished
2021-06-06 22:59:41,718 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(97d6f08d-050b-4117-a334-72ecace58929, queue=awx_private_queue)
[2021-06-06 22:59:41,718: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(97d6f08d-050b-4117-a334-72ecace58929, queue=awx_private_queue)
2021-06-06 22:59:41,746 DEBUG awx.main.dispatch delivered 97d6f08d-050b-4117-a334-72ecace58929 to worker[183] qsize 0
2021-06-06 22:59:41,749 DEBUG awx.main.dispatch task 97d6f08d-050b-4117-a334-72ecace58929 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 22:59:41,760 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 22:59:41,764 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:59:11.780781+00:00
2021-06-06 22:59:41,778 DEBUG awx.main.dispatch task 97d6f08d-050b-4117-a334-72ecace58929 is finished
2021-06-06 22:59:51,733 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1c286d0e-0be4-406e-9915-cc84b5433af8, queue=awx_private_queue)
[2021-06-06 22:59:51,733: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1c286d0e-0be4-406e-9915-cc84b5433af8, queue=awx_private_queue)
2021-06-06 22:59:51,750 DEBUG awx.main.dispatch delivered 1c286d0e-0be4-406e-9915-cc84b5433af8 to worker[182] qsize 0
2021-06-06 22:59:51,753 DEBUG awx.main.dispatch task 1c286d0e-0be4-406e-9915-cc84b5433af8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 22:59:51,754 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 22:59:51,771 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 22:59:51,802 DEBUG awx.main.dispatch task 1c286d0e-0be4-406e-9915-cc84b5433af8 is finished
RESULT 2
OKREADY
2021-06-06 23:00:11,467 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(170a0f2e-d210-41b0-9c66-bf9e2c8e8af7, queue=dev-awx-0)
[2021-06-06 23:00:11,467: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(170a0f2e-d210-41b0-9c66-bf9e2c8e8af7, queue=dev-awx-0)
2021-06-06 23:00:11,499 DEBUG awx.main.dispatch delivered 170a0f2e-d210-41b0-9c66-bf9e2c8e8af7 to worker[184] qsize 0
2021-06-06 23:00:11,501 DEBUG awx.main.dispatch task 170a0f2e-d210-41b0-9c66-bf9e2c8e8af7 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:00:11,504 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:00:11,525 DEBUG awx.main.dispatch task 170a0f2e-d210-41b0-9c66-bf9e2c8e8af7 is finished
2021-06-06 23:00:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(fc36027d-f054-46be-9251-bd9b237c09ba, queue=dev-awx-0)
[2021-06-06 23:00:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(fc36027d-f054-46be-9251-bd9b237c09ba, queue=dev-awx-0)
2021-06-06 23:00:11,647 DEBUG awx.main.dispatch delivered fc36027d-f054-46be-9251-bd9b237c09ba to worker[181] qsize 0
2021-06-06 23:00:11,649 DEBUG awx.main.dispatch task fc36027d-f054-46be-9251-bd9b237c09ba starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:00:11,661 DEBUG awx.main.dispatch task fc36027d-f054-46be-9251-bd9b237c09ba is finished
2021-06-06 23:00:11,718 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(93a95339-02f2-49e0-9d2c-19ca87babb40, queue=awx_private_queue)
[2021-06-06 23:00:11,718: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(93a95339-02f2-49e0-9d2c-19ca87babb40, queue=awx_private_queue)
2021-06-06 23:00:11,732 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9698e2bc-892b-4804-af14-3b1094f2d9da, queue=awx_private_queue)
2021-06-06 23:00:11,731 DEBUG awx.main.dispatch delivered 93a95339-02f2-49e0-9d2c-19ca87babb40 to worker[182] qsize 0
[2021-06-06 23:00:11,732: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9698e2bc-892b-4804-af14-3b1094f2d9da, queue=awx_private_queue)
2021-06-06 23:00:11,734 DEBUG awx.main.dispatch task 93a95339-02f2-49e0-9d2c-19ca87babb40 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:00:11,743 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:00:11,744 DEBUG awx.main.dispatch delivered 9698e2bc-892b-4804-af14-3b1094f2d9da to worker[184] qsize 0
2021-06-06 23:00:11,745 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 22:59:41.761004+00:00
2021-06-06 23:00:11,746 DEBUG awx.main.dispatch task 9698e2bc-892b-4804-af14-3b1094f2d9da starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:00:11,747 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:00:11,756 DEBUG awx.main.dispatch task 93a95339-02f2-49e0-9d2c-19ca87babb40 is finished
2021-06-06 23:00:11,761 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:00:11,785 DEBUG awx.main.dispatch task 9698e2bc-892b-4804-af14-3b1094f2d9da is finished
2021-06-06 23:00:31,745 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1c65cac7-5def-4ce7-9843-a848fdcb950b, queue=awx_private_queue)
[2021-06-06 23:00:31,745: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1c65cac7-5def-4ce7-9843-a848fdcb950b, queue=awx_private_queue)
2021-06-06 23:00:31,758 DEBUG awx.main.dispatch delivered 1c65cac7-5def-4ce7-9843-a848fdcb950b to worker[184] qsize 0
2021-06-06 23:00:31,760 DEBUG awx.main.dispatch task 1c65cac7-5def-4ce7-9843-a848fdcb950b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:00:31,762 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:00:31,778 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:00:31,811 DEBUG awx.main.dispatch task 1c65cac7-5def-4ce7-9843-a848fdcb950b is finished
2021-06-06 23:00:41,721 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(08f60fe9-6f51-45bd-89af-a9ac10cbde68, queue=awx_private_queue)
[2021-06-06 23:00:41,721: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(08f60fe9-6f51-45bd-89af-a9ac10cbde68, queue=awx_private_queue)
2021-06-06 23:00:41,754 DEBUG awx.main.dispatch delivered 08f60fe9-6f51-45bd-89af-a9ac10cbde68 to worker[181] qsize 0
2021-06-06 23:00:41,925 DEBUG awx.main.dispatch task 08f60fe9-6f51-45bd-89af-a9ac10cbde68 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:00:41,940 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:00:41,944 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:00:11.743222+00:00
2021-06-06 23:00:41,958 DEBUG awx.main.dispatch task 08f60fe9-6f51-45bd-89af-a9ac10cbde68 is finished
2021-06-06 23:00:51,747 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5ce9fa41-1d1f-439e-a62b-47033af6c20a, queue=awx_private_queue)
[2021-06-06 23:00:51,747: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5ce9fa41-1d1f-439e-a62b-47033af6c20a, queue=awx_private_queue)
2021-06-06 23:00:51,769 DEBUG awx.main.dispatch delivered 5ce9fa41-1d1f-439e-a62b-47033af6c20a to worker[184] qsize 0
2021-06-06 23:00:51,774 DEBUG awx.main.dispatch task 5ce9fa41-1d1f-439e-a62b-47033af6c20a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:00:51,778 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:00:51,799 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:00:51,829 DEBUG awx.main.dispatch task 5ce9fa41-1d1f-439e-a62b-47033af6c20a is finished
RESULT 2
OKREADY
2021-06-06 23:01:11,476 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c52af63e-7dd1-4d19-ac51-f4d349d45614, queue=dev-awx-0)
[2021-06-06 23:01:11,476: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c52af63e-7dd1-4d19-ac51-f4d349d45614, queue=dev-awx-0)
2021-06-06 23:01:11,502 DEBUG awx.main.dispatch delivered c52af63e-7dd1-4d19-ac51-f4d349d45614 to worker[183] qsize 0
2021-06-06 23:01:11,505 DEBUG awx.main.dispatch task c52af63e-7dd1-4d19-ac51-f4d349d45614 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:01:11,510 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:01:11,542 DEBUG awx.main.dispatch task c52af63e-7dd1-4d19-ac51-f4d349d45614 is finished
2021-06-06 23:01:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(df5cac8b-157f-48a3-bfd9-3ff5523d88c4, queue=dev-awx-0)
[2021-06-06 23:01:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(df5cac8b-157f-48a3-bfd9-3ff5523d88c4, queue=dev-awx-0)
2021-06-06 23:01:11,646 DEBUG awx.main.dispatch delivered df5cac8b-157f-48a3-bfd9-3ff5523d88c4 to worker[184] qsize 0
2021-06-06 23:01:11,648 DEBUG awx.main.dispatch task df5cac8b-157f-48a3-bfd9-3ff5523d88c4 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:01:11,663 DEBUG awx.main.dispatch task df5cac8b-157f-48a3-bfd9-3ff5523d88c4 is finished
2021-06-06 23:01:11,718 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f4472779-4053-4b55-a8e8-6a6a81b66f3c, queue=awx_private_queue)
[2021-06-06 23:01:11,718: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f4472779-4053-4b55-a8e8-6a6a81b66f3c, queue=awx_private_queue)
2021-06-06 23:01:11,733 DEBUG awx.main.dispatch delivered f4472779-4053-4b55-a8e8-6a6a81b66f3c to worker[184] qsize 0
2021-06-06 23:01:11,735 DEBUG awx.main.dispatch task f4472779-4053-4b55-a8e8-6a6a81b66f3c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:01:11,743 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:01:11,745 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(eddf7484-497f-4d16-8ade-2746ce8e4941, queue=awx_private_queue)
[2021-06-06 23:01:11,745: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(eddf7484-497f-4d16-8ade-2746ce8e4941, queue=awx_private_queue)
2021-06-06 23:01:11,747 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:00:41.940466+00:00
2021-06-06 23:01:11,759 DEBUG awx.main.dispatch task f4472779-4053-4b55-a8e8-6a6a81b66f3c is finished
2021-06-06 23:01:11,758 DEBUG awx.main.dispatch delivered eddf7484-497f-4d16-8ade-2746ce8e4941 to worker[181] qsize 0
2021-06-06 23:01:11,761 DEBUG awx.main.dispatch task eddf7484-497f-4d16-8ade-2746ce8e4941 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:01:11,763 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:01:11,778 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:01:11,823 DEBUG awx.main.dispatch task eddf7484-497f-4d16-8ade-2746ce8e4941 is finished
2021-06-06 23:01:31,755 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6076acf9-0d71-4eef-a66c-154d7a66e39b, queue=awx_private_queue)
[2021-06-06 23:01:31,755: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6076acf9-0d71-4eef-a66c-154d7a66e39b, queue=awx_private_queue)
2021-06-06 23:01:31,769 DEBUG awx.main.dispatch delivered 6076acf9-0d71-4eef-a66c-154d7a66e39b to worker[183] qsize 0
2021-06-06 23:01:31,771 DEBUG awx.main.dispatch task 6076acf9-0d71-4eef-a66c-154d7a66e39b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:01:31,772 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:01:31,789 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:01:31,821 DEBUG awx.main.dispatch task 6076acf9-0d71-4eef-a66c-154d7a66e39b is finished
2021-06-06 23:01:41,718 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7c39b92a-71b0-434f-b6bb-142b3257c4df, queue=awx_private_queue)
[2021-06-06 23:01:41,718: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7c39b92a-71b0-434f-b6bb-142b3257c4df, queue=awx_private_queue)
2021-06-06 23:01:41,743 DEBUG awx.main.dispatch delivered 7c39b92a-71b0-434f-b6bb-142b3257c4df to worker[182] qsize 0
2021-06-06 23:01:41,746 DEBUG awx.main.dispatch task 7c39b92a-71b0-434f-b6bb-142b3257c4df starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:01:41,922 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:01:41,924 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:01:11.743995+00:00
2021-06-06 23:01:41,941 DEBUG awx.main.dispatch task 7c39b92a-71b0-434f-b6bb-142b3257c4df is finished
2021-06-06 23:01:51,755 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8d668fb8-ebd5-4bdb-b828-f9a479f673de, queue=awx_private_queue)
[2021-06-06 23:01:51,755: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8d668fb8-ebd5-4bdb-b828-f9a479f673de, queue=awx_private_queue)
2021-06-06 23:01:51,769 DEBUG awx.main.dispatch delivered 8d668fb8-ebd5-4bdb-b828-f9a479f673de to worker[183] qsize 0
2021-06-06 23:01:51,772 DEBUG awx.main.dispatch task 8d668fb8-ebd5-4bdb-b828-f9a479f673de starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:01:51,773 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:01:51,789 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:01:51,811 DEBUG awx.main.dispatch task 8d668fb8-ebd5-4bdb-b828-f9a479f673de is finished
RESULT 2
OKREADY
2021-06-06 23:02:11,491 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(13875ec4-86aa-44f0-845e-132f1e68639d, queue=dev-awx-0)
[2021-06-06 23:02:11,491: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(13875ec4-86aa-44f0-845e-132f1e68639d, queue=dev-awx-0)
2021-06-06 23:02:11,556 DEBUG awx.main.dispatch delivered 13875ec4-86aa-44f0-845e-132f1e68639d to worker[184] qsize 0
2021-06-06 23:02:11,561 DEBUG awx.main.dispatch task 13875ec4-86aa-44f0-845e-132f1e68639d starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:02:11,565 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:02:11,612 DEBUG awx.main.dispatch task 13875ec4-86aa-44f0-845e-132f1e68639d is finished
2021-06-06 23:02:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(e0bf4aae-841f-46a4-a134-5c7aef11b3df, queue=dev-awx-0)
[2021-06-06 23:02:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(e0bf4aae-841f-46a4-a134-5c7aef11b3df, queue=dev-awx-0)
2021-06-06 23:02:11,648 DEBUG awx.main.dispatch delivered e0bf4aae-841f-46a4-a134-5c7aef11b3df to worker[181] qsize 0
2021-06-06 23:02:11,651 DEBUG awx.main.dispatch task e0bf4aae-841f-46a4-a134-5c7aef11b3df starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:02:11,718 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c02c0d3a-fe81-4382-9f95-02b949f823b8, queue=awx_private_queue)
[2021-06-06 23:02:11,718: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c02c0d3a-fe81-4382-9f95-02b949f823b8, queue=awx_private_queue)
2021-06-06 23:02:11,733 DEBUG awx.main.dispatch delivered c02c0d3a-fe81-4382-9f95-02b949f823b8 to worker[184] qsize 0
2021-06-06 23:02:11,736 DEBUG awx.main.dispatch task c02c0d3a-fe81-4382-9f95-02b949f823b8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:02:11,755 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(17e3b8cc-2347-401b-b9a4-752a39ba4938, queue=awx_private_queue)
[2021-06-06 23:02:11,755: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(17e3b8cc-2347-401b-b9a4-752a39ba4938, queue=awx_private_queue)
2021-06-06 23:02:11,768 DEBUG awx.main.dispatch delivered 17e3b8cc-2347-401b-b9a4-752a39ba4938 to worker[182] qsize 0
2021-06-06 23:02:11,772 DEBUG awx.main.dispatch task 17e3b8cc-2347-401b-b9a4-752a39ba4938 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:02:11,775 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:02:11,780 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:02:11,789 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:01:41.922896+00:00
2021-06-06 23:02:11,790 DEBUG awx.main.dispatch task e0bf4aae-841f-46a4-a134-5c7aef11b3df is finished
2021-06-06 23:02:11,868 DEBUG awx.main.dispatch task c02c0d3a-fe81-4382-9f95-02b949f823b8 is finished
2021-06-06 23:02:11,987 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:02:12,031 DEBUG awx.main.dispatch task 17e3b8cc-2347-401b-b9a4-752a39ba4938 is finished
2021-06-06 23:02:31,762 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(706bd456-5e4b-48e0-811e-dcd512da674d, queue=awx_private_queue)
[2021-06-06 23:02:31,762: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(706bd456-5e4b-48e0-811e-dcd512da674d, queue=awx_private_queue)
2021-06-06 23:02:31,776 DEBUG awx.main.dispatch delivered 706bd456-5e4b-48e0-811e-dcd512da674d to worker[181] qsize 0
2021-06-06 23:02:31,778 DEBUG awx.main.dispatch task 706bd456-5e4b-48e0-811e-dcd512da674d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:02:31,780 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:02:31,805 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:02:31,842 DEBUG awx.main.dispatch task 706bd456-5e4b-48e0-811e-dcd512da674d is finished
2021-06-06 23:02:41,718 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d6254177-69f1-4125-8e5a-907a65795ee2, queue=awx_private_queue)
[2021-06-06 23:02:41,718: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d6254177-69f1-4125-8e5a-907a65795ee2, queue=awx_private_queue)
2021-06-06 23:02:41,744 DEBUG awx.main.dispatch delivered d6254177-69f1-4125-8e5a-907a65795ee2 to worker[182] qsize 0
2021-06-06 23:02:41,746 DEBUG awx.main.dispatch task d6254177-69f1-4125-8e5a-907a65795ee2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:02:41,754 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:02:41,756 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:02:11.781014+00:00
2021-06-06 23:02:41,768 DEBUG awx.main.dispatch task d6254177-69f1-4125-8e5a-907a65795ee2 is finished
2021-06-06 23:02:51,763 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b82aa16e-0e23-436f-8f88-7c3e6f7d761b, queue=awx_private_queue)
[2021-06-06 23:02:51,763: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b82aa16e-0e23-436f-8f88-7c3e6f7d761b, queue=awx_private_queue)
2021-06-06 23:02:51,788 DEBUG awx.main.dispatch delivered b82aa16e-0e23-436f-8f88-7c3e6f7d761b to worker[182] qsize 0
2021-06-06 23:02:51,790 DEBUG awx.main.dispatch task b82aa16e-0e23-436f-8f88-7c3e6f7d761b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:02:51,792 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:02:51,806 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:02:51,838 DEBUG awx.main.dispatch task b82aa16e-0e23-436f-8f88-7c3e6f7d761b is finished
RESULT 2
OKREADY
2021-06-06 23:03:11,502 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(2da8da88-320d-45dc-8a97-63b1a04c5b26, queue=dev-awx-0)
[2021-06-06 23:03:11,502: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(2da8da88-320d-45dc-8a97-63b1a04c5b26, queue=dev-awx-0)
2021-06-06 23:03:11,550 DEBUG awx.main.dispatch delivered 2da8da88-320d-45dc-8a97-63b1a04c5b26 to worker[181] qsize 0
2021-06-06 23:03:11,555 DEBUG awx.main.dispatch task 2da8da88-320d-45dc-8a97-63b1a04c5b26 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:03:11,558 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:03:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c58401b8-4934-46c6-8bae-df847b64ef07, queue=dev-awx-0)
[2021-06-06 23:03:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c58401b8-4934-46c6-8bae-df847b64ef07, queue=dev-awx-0)
2021-06-06 23:03:11,637 DEBUG awx.main.dispatch task 2da8da88-320d-45dc-8a97-63b1a04c5b26 is finished
2021-06-06 23:03:11,647 DEBUG awx.main.dispatch delivered c58401b8-4934-46c6-8bae-df847b64ef07 to worker[184] qsize 0
2021-06-06 23:03:11,650 DEBUG awx.main.dispatch task c58401b8-4934-46c6-8bae-df847b64ef07 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:03:11,669 DEBUG awx.main.dispatch task c58401b8-4934-46c6-8bae-df847b64ef07 is finished
2021-06-06 23:03:11,718 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(98d1468e-a097-4f64-8f1b-6d44ac0f1378, queue=awx_private_queue)
[2021-06-06 23:03:11,718: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(98d1468e-a097-4f64-8f1b-6d44ac0f1378, queue=awx_private_queue)
2021-06-06 23:03:11,732 DEBUG awx.main.dispatch delivered 98d1468e-a097-4f64-8f1b-6d44ac0f1378 to worker[182] qsize 0
2021-06-06 23:03:11,735 DEBUG awx.main.dispatch task 98d1468e-a097-4f64-8f1b-6d44ac0f1378 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:03:11,754 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:03:11,758 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:02:41.754353+00:00
2021-06-06 23:03:11,762 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(42647067-0796-48d0-9eda-2553a1982007, queue=awx_private_queue)
[2021-06-06 23:03:11,762: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(42647067-0796-48d0-9eda-2553a1982007, queue=awx_private_queue)
2021-06-06 23:03:11,771 DEBUG awx.main.dispatch task 98d1468e-a097-4f64-8f1b-6d44ac0f1378 is finished
2021-06-06 23:03:11,776 DEBUG awx.main.dispatch delivered 42647067-0796-48d0-9eda-2553a1982007 to worker[184] qsize 0
2021-06-06 23:03:11,777 DEBUG awx.main.dispatch task 42647067-0796-48d0-9eda-2553a1982007 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:03:11,779 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:03:11,795 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:03:11,826 DEBUG awx.main.dispatch task 42647067-0796-48d0-9eda-2553a1982007 is finished
2021-06-06 23:03:31,778 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(109602d3-3196-479a-91a9-6df0d54c7d1c, queue=awx_private_queue)
[2021-06-06 23:03:31,778: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(109602d3-3196-479a-91a9-6df0d54c7d1c, queue=awx_private_queue)
2021-06-06 23:03:31,791 DEBUG awx.main.dispatch delivered 109602d3-3196-479a-91a9-6df0d54c7d1c to worker[183] qsize 0
2021-06-06 23:03:31,795 DEBUG awx.main.dispatch task 109602d3-3196-479a-91a9-6df0d54c7d1c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:03:31,799 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:03:32,215 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:03:32,252 DEBUG awx.main.dispatch task 109602d3-3196-479a-91a9-6df0d54c7d1c is finished
2021-06-06 23:03:41,724 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8a5386c3-3f2e-4b1c-b64c-bcd38e03b0d2, queue=awx_private_queue)
[2021-06-06 23:03:41,724: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8a5386c3-3f2e-4b1c-b64c-bcd38e03b0d2, queue=awx_private_queue)
2021-06-06 23:03:43,837 DEBUG awx.main.dispatch delivered 8a5386c3-3f2e-4b1c-b64c-bcd38e03b0d2 to worker[183] qsize 0
2021-06-06 23:03:43,845 DEBUG awx.main.dispatch task 8a5386c3-3f2e-4b1c-b64c-bcd38e03b0d2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:03:43,875 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:03:43,894 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:03:11.754360+00:00
2021-06-06 23:03:43,994 DEBUG awx.main.dispatch task 8a5386c3-3f2e-4b1c-b64c-bcd38e03b0d2 is finished
2021-06-06 23:03:51,778 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ef886339-f8ff-495e-8a33-f03ba17b88dd, queue=awx_private_queue)
[2021-06-06 23:03:51,778: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ef886339-f8ff-495e-8a33-f03ba17b88dd, queue=awx_private_queue)
2021-06-06 23:03:51,792 DEBUG awx.main.dispatch delivered ef886339-f8ff-495e-8a33-f03ba17b88dd to worker[184] qsize 0
2021-06-06 23:03:52,889 DEBUG awx.main.dispatch task ef886339-f8ff-495e-8a33-f03ba17b88dd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:03:52,894 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:03:54,796 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:03:55,600 DEBUG awx.main.dispatch task ef886339-f8ff-495e-8a33-f03ba17b88dd is finished
RESULT 2
OKREADY
2021-06-06 23:04:11,503 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(98456910-fddb-4560-8ffe-6702f65c7aa5, queue=dev-awx-0)
[2021-06-06 23:04:11,503: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(98456910-fddb-4560-8ffe-6702f65c7aa5, queue=dev-awx-0)
2021-06-06 23:04:11,580 DEBUG awx.main.dispatch delivered 98456910-fddb-4560-8ffe-6702f65c7aa5 to worker[181] qsize 0
2021-06-06 23:04:11,587 DEBUG awx.main.dispatch task 98456910-fddb-4560-8ffe-6702f65c7aa5 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:04:11,593 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:04:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(876af89f-7200-41ed-99b1-aba567749deb, queue=dev-awx-0)
[2021-06-06 23:04:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(876af89f-7200-41ed-99b1-aba567749deb, queue=dev-awx-0)
2021-06-06 23:04:11,635 DEBUG awx.main.dispatch task 98456910-fddb-4560-8ffe-6702f65c7aa5 is finished
2021-06-06 23:04:11,647 DEBUG awx.main.dispatch delivered 876af89f-7200-41ed-99b1-aba567749deb to worker[182] qsize 0
2021-06-06 23:04:11,651 DEBUG awx.main.dispatch task 876af89f-7200-41ed-99b1-aba567749deb starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:04:11,676 DEBUG awx.main.dispatch task 876af89f-7200-41ed-99b1-aba567749deb is finished
2021-06-06 23:04:11,777 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(52ead1ea-d761-4258-b972-a5424153eaf7, queue=awx_private_queue)
[2021-06-06 23:04:11,777: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(52ead1ea-d761-4258-b972-a5424153eaf7, queue=awx_private_queue)
2021-06-06 23:04:11,791 DEBUG awx.main.dispatch delivered 52ead1ea-d761-4258-b972-a5424153eaf7 to worker[183] qsize 0
2021-06-06 23:04:11,794 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(76458102-ef8b-482a-a5cf-14e572e79f4b, queue=awx_private_queue)
[2021-06-06 23:04:11,794: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(76458102-ef8b-482a-a5cf-14e572e79f4b, queue=awx_private_queue)
2021-06-06 23:04:11,795 DEBUG awx.main.dispatch task 52ead1ea-d761-4258-b972-a5424153eaf7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:04:11,801 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:04:11,807 DEBUG awx.main.dispatch delivered 76458102-ef8b-482a-a5cf-14e572e79f4b to worker[184] qsize 0
2021-06-06 23:04:11,812 DEBUG awx.main.dispatch task 76458102-ef8b-482a-a5cf-14e572e79f4b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:04:11,827 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:04:11,830 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:04:11,837 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:03:43.876282+00:00
2021-06-06 23:04:11,855 DEBUG awx.main.dispatch task 76458102-ef8b-482a-a5cf-14e572e79f4b is finished
2021-06-06 23:04:11,876 DEBUG awx.main.dispatch task 52ead1ea-d761-4258-b972-a5424153eaf7 is finished
2021-06-06 23:04:31,778 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5b02dd64-5ff1-412a-8414-8e3d3de583ea, queue=awx_private_queue)
[2021-06-06 23:04:31,778: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5b02dd64-5ff1-412a-8414-8e3d3de583ea, queue=awx_private_queue)
2021-06-06 23:04:31,798 DEBUG awx.main.dispatch delivered 5b02dd64-5ff1-412a-8414-8e3d3de583ea to worker[182] qsize 0
2021-06-06 23:04:31,801 DEBUG awx.main.dispatch task 5b02dd64-5ff1-412a-8414-8e3d3de583ea starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:04:31,802 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:04:32,119 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:04:32,189 DEBUG awx.main.dispatch task 5b02dd64-5ff1-412a-8414-8e3d3de583ea is finished
2021-06-06 23:04:41,794 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d5e37f3f-f47c-48ee-bdcf-57ae9eecbc0a, queue=awx_private_queue)
[2021-06-06 23:04:41,794: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d5e37f3f-f47c-48ee-bdcf-57ae9eecbc0a, queue=awx_private_queue)
2021-06-06 23:04:41,812 DEBUG awx.main.dispatch delivered d5e37f3f-f47c-48ee-bdcf-57ae9eecbc0a to worker[184] qsize 0
2021-06-06 23:04:41,814 DEBUG awx.main.dispatch task d5e37f3f-f47c-48ee-bdcf-57ae9eecbc0a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:04:51,778 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3a8af997-cd18-4b7a-8732-b6ac9f739cd2, queue=awx_private_queue)
[2021-06-06 23:04:51,778: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3a8af997-cd18-4b7a-8732-b6ac9f739cd2, queue=awx_private_queue)
2021-06-06 23:04:51,801 DEBUG awx.main.dispatch delivered 3a8af997-cd18-4b7a-8732-b6ac9f739cd2 to worker[182] qsize 0
2021-06-06 23:04:51,805 DEBUG awx.main.dispatch task 3a8af997-cd18-4b7a-8732-b6ac9f739cd2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:04:51,806 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:04:51,982 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:04:53,425 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:04:11.831235+00:00
2021-06-06 23:04:55,910 DEBUG awx.main.dispatch task d5e37f3f-f47c-48ee-bdcf-57ae9eecbc0a is finished
RESULT 2
OKREADY
2021-06-06 23:05:06,572 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:05:06,620 DEBUG awx.main.dispatch task 3a8af997-cd18-4b7a-8732-b6ac9f739cd2 is finished
2021-06-06 23:05:11,506 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(65842f95-a7e6-4019-9dd4-53b23000e90d, queue=dev-awx-0)
[2021-06-06 23:05:11,506: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(65842f95-a7e6-4019-9dd4-53b23000e90d, queue=dev-awx-0)
2021-06-06 23:05:11,635 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(abe47d20-221d-4a9e-ae2a-d6ef29d946fe, queue=dev-awx-0)
[2021-06-06 23:05:11,635: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(abe47d20-221d-4a9e-ae2a-d6ef29d946fe, queue=dev-awx-0)
2021-06-06 23:05:11,676 DEBUG awx.main.dispatch delivered 65842f95-a7e6-4019-9dd4-53b23000e90d to worker[184] qsize 0
2021-06-06 23:05:11,678 DEBUG awx.main.dispatch delivered abe47d20-221d-4a9e-ae2a-d6ef29d946fe to worker[181] qsize 0
2021-06-06 23:05:11,678 DEBUG awx.main.dispatch task 65842f95-a7e6-4019-9dd4-53b23000e90d starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:05:11,680 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:05:11,679 DEBUG awx.main.dispatch task abe47d20-221d-4a9e-ae2a-d6ef29d946fe starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:05:14,358 DEBUG awx.main.dispatch task abe47d20-221d-4a9e-ae2a-d6ef29d946fe is finished
2021-06-06 23:05:14,525 DEBUG awx.main.dispatch task 65842f95-a7e6-4019-9dd4-53b23000e90d is finished
2021-06-06 23:05:11,778 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(034ee181-24ee-48f9-b5be-378cfc005454, queue=awx_private_queue)
[2021-06-06 23:05:11,778: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(034ee181-24ee-48f9-b5be-378cfc005454, queue=awx_private_queue)
2021-06-06 23:05:14,577 DEBUG awx.main.dispatch delivered 034ee181-24ee-48f9-b5be-378cfc005454 to worker[182] qsize 0
2021-06-06 23:05:14,578 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ee4f08cb-69fe-4327-b27e-82ad70279184, queue=awx_private_queue)
[2021-06-06 23:05:14,578: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ee4f08cb-69fe-4327-b27e-82ad70279184, queue=awx_private_queue)
2021-06-06 23:05:14,579 DEBUG awx.main.dispatch task 034ee181-24ee-48f9-b5be-378cfc005454 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:05:14,580 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:05:14,598 DEBUG awx.main.dispatch delivered ee4f08cb-69fe-4327-b27e-82ad70279184 to worker[181] qsize 0
2021-06-06 23:05:14,600 DEBUG awx.main.dispatch task ee4f08cb-69fe-4327-b27e-82ad70279184 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:05:14,619 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:05:14,755 DEBUG awx.main.dispatch task 034ee181-24ee-48f9-b5be-378cfc005454 is finished
2021-06-06 23:05:14,980 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:05:14,985 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:04:51.982635+00:00
2021-06-06 23:05:16,018 DEBUG awx.main.dispatch task ee4f08cb-69fe-4327-b27e-82ad70279184 is finished
2021-06-06 23:05:31,783 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e9675035-ae96-4311-ba51-a58d5753f449, queue=awx_private_queue)
[2021-06-06 23:05:31,783: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e9675035-ae96-4311-ba51-a58d5753f449, queue=awx_private_queue)
2021-06-06 23:05:31,801 DEBUG awx.main.dispatch delivered e9675035-ae96-4311-ba51-a58d5753f449 to worker[182] qsize 0
2021-06-06 23:05:31,803 DEBUG awx.main.dispatch task e9675035-ae96-4311-ba51-a58d5753f449 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:05:31,804 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:05:44,285 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:05:44,578 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(72ff0125-4267-47b0-8b06-d400b2b059b8, queue=awx_private_queue)
[2021-06-06 23:05:44,578: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(72ff0125-4267-47b0-8b06-d400b2b059b8, queue=awx_private_queue)
2021-06-06 23:05:44,593 DEBUG awx.main.dispatch delivered 72ff0125-4267-47b0-8b06-d400b2b059b8 to worker[184] qsize 0
2021-06-06 23:05:44,595 DEBUG awx.main.dispatch task 72ff0125-4267-47b0-8b06-d400b2b059b8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:05:51,168 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:05:51,233 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:05:14.980728+00:00
2021-06-06 23:05:51,783 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b9574896-fc43-4e8d-8e4b-853649e95387, queue=awx_private_queue)
[2021-06-06 23:05:51,783: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b9574896-fc43-4e8d-8e4b-853649e95387, queue=awx_private_queue)
2021-06-06 23:05:51,797 DEBUG awx.main.dispatch delivered b9574896-fc43-4e8d-8e4b-853649e95387 to worker[183] qsize 0
2021-06-06 23:05:51,800 DEBUG awx.main.dispatch task b9574896-fc43-4e8d-8e4b-853649e95387 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:05:51,803 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:05:52,352 DEBUG awx.main.dispatch task e9675035-ae96-4311-ba51-a58d5753f449 is finished
2021-06-06 23:05:52,557 DEBUG awx.main.dispatch task 72ff0125-4267-47b0-8b06-d400b2b059b8 is finished
RESULT 2
OKREADY
2021-06-06 23:06:04,857 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:06:11,512 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(9f026ff8-3483-4a19-abac-dd2de52daeff, queue=dev-awx-0)
[2021-06-06 23:06:11,512: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(9f026ff8-3483-4a19-abac-dd2de52daeff, queue=dev-awx-0)
2021-06-06 23:06:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(dd57a562-6729-469e-bfba-19132af2d226, queue=dev-awx-0)
[2021-06-06 23:06:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(dd57a562-6729-469e-bfba-19132af2d226, queue=dev-awx-0)
2021-06-06 23:06:11,693 DEBUG awx.main.dispatch delivered 9f026ff8-3483-4a19-abac-dd2de52daeff to worker[182] qsize 0
2021-06-06 23:06:11,697 DEBUG awx.main.dispatch delivered dd57a562-6729-469e-bfba-19132af2d226 to worker[184] qsize 0
2021-06-06 23:06:11,698 DEBUG awx.main.dispatch task 9f026ff8-3483-4a19-abac-dd2de52daeff starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:06:11,701 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:06:11,700 DEBUG awx.main.dispatch task dd57a562-6729-469e-bfba-19132af2d226 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:06:11,783 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(dbd6382b-9ff5-4a42-ab81-f73d1a5bd1b0, queue=awx_private_queue)
[2021-06-06 23:06:11,783: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(dbd6382b-9ff5-4a42-ab81-f73d1a5bd1b0, queue=awx_private_queue)
2021-06-06 23:06:11,796 DEBUG awx.main.dispatch delivered dbd6382b-9ff5-4a42-ab81-f73d1a5bd1b0 to worker[181] qsize 0
2021-06-06 23:06:11,799 DEBUG awx.main.dispatch task dbd6382b-9ff5-4a42-ab81-f73d1a5bd1b0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:06:11,803 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:06:14,578 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(56053571-0ded-448f-9414-379f1becc51f, queue=awx_private_queue)
[2021-06-06 23:06:14,578: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(56053571-0ded-448f-9414-379f1becc51f, queue=awx_private_queue)
2021-06-06 23:06:19,627 WARNING awx.main.dispatch scaling up worker pid:193
2021-06-06 23:06:19,632 DEBUG awx.main.dispatch delivered 56053571-0ded-448f-9414-379f1becc51f to worker[193] qsize 0
2021-06-06 23:06:19,636 DEBUG awx.main.dispatch task 56053571-0ded-448f-9414-379f1becc51f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:06:20,050 DEBUG awx.main.dispatch task dd57a562-6729-469e-bfba-19132af2d226 is finished
2021-06-06 23:06:25,409 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:06:25,478 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:05:51.168828+00:00
2021-06-06 23:06:27,967 DEBUG awx.main.dispatch task 9f026ff8-3483-4a19-abac-dd2de52daeff is finished
2021-06-06 23:06:28,687 DEBUG awx.main.dispatch task 56053571-0ded-448f-9414-379f1becc51f is finished
2021-06-06 23:06:31,788 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(386d9ad5-1544-48c8-908e-d1f644706c98, queue=awx_private_queue)
[2021-06-06 23:06:31,788: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(386d9ad5-1544-48c8-908e-d1f644706c98, queue=awx_private_queue)
2021-06-06 23:06:31,805 DEBUG awx.main.dispatch delivered 386d9ad5-1544-48c8-908e-d1f644706c98 to worker[182] qsize 0
2021-06-06 23:06:31,808 DEBUG awx.main.dispatch task 386d9ad5-1544-48c8-908e-d1f644706c98 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:06:31,809 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:06:38,483 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-06-06 23:06:38,483 DEBUG awx.main.dispatch task dbd6382b-9ff5-4a42-ab81-f73d1a5bd1b0 is finished
2021-06-06 23:06:44,581 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c215e2c8-93a5-43b0-bf80-00ca4ab0ba8c, queue=awx_private_queue)
[2021-06-06 23:06:44,581: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c215e2c8-93a5-43b0-bf80-00ca4ab0ba8c, queue=awx_private_queue)
2021-06-06 23:06:44,595 DEBUG awx.main.dispatch delivered c215e2c8-93a5-43b0-bf80-00ca4ab0ba8c to worker[181] qsize 0
2021-06-06 23:06:44,598 DEBUG awx.main.dispatch task c215e2c8-93a5-43b0-bf80-00ca4ab0ba8c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:06:49,851 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:06:50,275 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:06:25.409963+00:00
2021-06-06 23:06:51,787 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6d68d6ad-096e-42a0-87a5-2989f9174f98, queue=awx_private_queue)
[2021-06-06 23:06:51,787: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6d68d6ad-096e-42a0-87a5-2989f9174f98, queue=awx_private_queue)
2021-06-06 23:06:51,802 DEBUG awx.main.dispatch delivered 6d68d6ad-096e-42a0-87a5-2989f9174f98 to worker[184] qsize 0
2021-06-06 23:06:51,808 DEBUG awx.main.dispatch task 6d68d6ad-096e-42a0-87a5-2989f9174f98 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:06:51,811 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:06:53,154 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-06-06 23:06:53,155 DEBUG awx.main.dispatch task 6d68d6ad-096e-42a0-87a5-2989f9174f98 is finished
2021-06-06 23:06:53,171 DEBUG awx.main.dispatch task c215e2c8-93a5-43b0-bf80-00ca4ab0ba8c is finished
2021-06-06 23:06:53,207 DEBUG awx.main.dispatch task b9574896-fc43-4e8d-8e4b-853649e95387 is finished
2021-06-06 23:06:53,792 DEBUG awx.main.scheduler Starting Scheduler
RESULT 2
OKREADY
2021-06-06 23:07:11,513 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(41279e8a-173c-49bf-bc57-e149c4dac89e, queue=dev-awx-0)
[2021-06-06 23:07:11,513: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(41279e8a-173c-49bf-bc57-e149c4dac89e, queue=dev-awx-0)
2021-06-06 23:07:11,530 WARNING awx.main.dispatch scaling down worker pid:181
2021-06-06 23:07:11,533 WARNING awx.main.dispatch worker exiting gracefully pid:181
2021-06-06 23:07:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(262723d0-ac99-40ca-b221-e580e59366f0, queue=dev-awx-0)
[2021-06-06 23:07:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(262723d0-ac99-40ca-b221-e580e59366f0, queue=dev-awx-0)
2021-06-06 23:07:11,787 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(eee6ffce-2b8f-4597-9e9c-b55e32c446da, queue=awx_private_queue)
[2021-06-06 23:07:11,787: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(eee6ffce-2b8f-4597-9e9c-b55e32c446da, queue=awx_private_queue)
2021-06-06 23:07:14,581 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c17e4df2-a2b1-4ba9-afcf-c52fe032b7d7, queue=awx_private_queue)
[2021-06-06 23:07:14,581: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c17e4df2-a2b1-4ba9-afcf-c52fe032b7d7, queue=awx_private_queue)
2021-06-06 23:07:31,795 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2d411362-1940-476d-9a3e-d2a06ca6cb41, queue=awx_private_queue)
[2021-06-06 23:07:31,795: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2d411362-1940-476d-9a3e-d2a06ca6cb41, queue=awx_private_queue)
2021-06-06 23:07:38,819 DEBUG awx.main.dispatch delivered 41279e8a-173c-49bf-bc57-e149c4dac89e to worker[193] qsize 0
2021-06-06 23:07:39,766 DEBUG awx.main.dispatch delivered 262723d0-ac99-40ca-b221-e580e59366f0 to worker[184] qsize 0
2021-06-06 23:07:39,767 DEBUG awx.main.dispatch task 41279e8a-173c-49bf-bc57-e149c4dac89e starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:07:39,768 DEBUG awx.main.dispatch task 262723d0-ac99-40ca-b221-e580e59366f0 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:07:39,771 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:07:39,770 DEBUG awx.main.dispatch delivered eee6ffce-2b8f-4597-9e9c-b55e32c446da to worker[183] qsize 0
2021-06-06 23:07:39,772 DEBUG awx.main.dispatch task eee6ffce-2b8f-4597-9e9c-b55e32c446da starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:07:39,774 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:07:39,780 WARNING awx.main.dispatch scaling up worker pid:196
2021-06-06 23:07:39,785 DEBUG awx.main.dispatch delivered c17e4df2-a2b1-4ba9-afcf-c52fe032b7d7 to worker[196] qsize 0
2021-06-06 23:07:39,789 DEBUG awx.main.dispatch task c17e4df2-a2b1-4ba9-afcf-c52fe032b7d7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:07:39,795 WARNING awx.main.dispatch scaling up worker pid:198
2021-06-06 23:07:39,800 DEBUG awx.main.dispatch delivered 2d411362-1940-476d-9a3e-d2a06ca6cb41 to worker[198] qsize 0
2021-06-06 23:07:39,806 DEBUG awx.main.dispatch task 2d411362-1940-476d-9a3e-d2a06ca6cb41 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:07:39,810 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:07:44,581 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(26067bdb-e8fd-44bb-9aae-231df4c73f2f, queue=awx_private_queue)
[2021-06-06 23:07:44,581: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(26067bdb-e8fd-44bb-9aae-231df4c73f2f, queue=awx_private_queue)
2021-06-06 23:07:44,616 WARNING awx.main.dispatch scaling up worker pid:200
2021-06-06 23:07:44,621 DEBUG awx.main.dispatch delivered 26067bdb-e8fd-44bb-9aae-231df4c73f2f to worker[200] qsize 0
2021-06-06 23:07:44,625 DEBUG awx.main.dispatch task 26067bdb-e8fd-44bb-9aae-231df4c73f2f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:07:48,600 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:07:48,648 DEBUG awx.main.dispatch task 386d9ad5-1544-48c8-908e-d1f644706c98 is finished
2021-06-06 23:07:48,767 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:06:49.851880+00:00
2021-06-06 23:07:49,748 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:07:51,275 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-06-06 23:07:51,276 DEBUG awx.main.dispatch task 2d411362-1940-476d-9a3e-d2a06ca6cb41 is finished
2021-06-06 23:07:51,797 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8d01bfb7-3cb2-49a9-8019-62adc8abf4be, queue=awx_private_queue)
[2021-06-06 23:07:51,797: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8d01bfb7-3cb2-49a9-8019-62adc8abf4be, queue=awx_private_queue)
2021-06-06 23:07:51,817 DEBUG awx.main.dispatch delivered 8d01bfb7-3cb2-49a9-8019-62adc8abf4be to worker[198] qsize 0
2021-06-06 23:07:51,821 DEBUG awx.main.dispatch task 8d01bfb7-3cb2-49a9-8019-62adc8abf4be starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:07:51,825 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:07:52,297 DEBUG awx.main.dispatch task 262723d0-ac99-40ca-b221-e580e59366f0 is finished
2021-06-06 23:07:52,445 DEBUG awx.main.dispatch task eee6ffce-2b8f-4597-9e9c-b55e32c446da is finished
RESULT 2
OKREADY
2021-06-06 23:08:11,396 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(69cbf4c5-3208-4241-baf4-51b68abe4a2c, queue=dev-awx-0)
[2021-06-06 23:08:11,396: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(69cbf4c5-3208-4241-baf4-51b68abe4a2c, queue=dev-awx-0)
2021-06-06 23:08:11,416 DEBUG awx.main.dispatch delivered 69cbf4c5-3208-4241-baf4-51b68abe4a2c to worker[182] qsize 0
2021-06-06 23:08:11,421 DEBUG awx.main.dispatch task 69cbf4c5-3208-4241-baf4-51b68abe4a2c starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-06-06 23:08:11,432 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-06-06 23:08:11,513 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d95d89f2-0dc2-4e81-9ac6-0c21655aaa0a, queue=dev-awx-0)
[2021-06-06 23:08:11,513: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d95d89f2-0dc2-4e81-9ac6-0c21655aaa0a, queue=dev-awx-0)
2021-06-06 23:08:11,528 WARNING awx.main.dispatch scaling down worker pid:183
2021-06-06 23:08:11,529 WARNING awx.main.dispatch scaling down worker pid:184
2021-06-06 23:08:11,531 WARNING awx.main.dispatch worker exiting gracefully pid:183
2021-06-06 23:08:11,537 WARNING awx.main.dispatch worker exiting gracefully pid:184
2021-06-06 23:08:11,635 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(4758983f-227c-4099-a09e-5c83835a44ca, queue=dev-awx-0)
[2021-06-06 23:08:11,635: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(4758983f-227c-4099-a09e-5c83835a44ca, queue=dev-awx-0)
2021-06-06 23:08:11,795 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8db2d90c-4d83-4d34-9855-b76c33194441, queue=awx_private_queue)
[2021-06-06 23:08:11,795: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8db2d90c-4d83-4d34-9855-b76c33194441, queue=awx_private_queue)
2021-06-06 23:08:14,582 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8e890b04-0477-493b-bb4a-869d99dc5573, queue=awx_private_queue)
[2021-06-06 23:08:14,582: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8e890b04-0477-493b-bb4a-869d99dc5573, queue=awx_private_queue)
2021-06-06 23:08:16,872 DEBUG awx.main.dispatch task c17e4df2-a2b1-4ba9-afcf-c52fe032b7d7 is finished
2021-06-06 23:08:17,410 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:08:21,540 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:08:21,626 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:07:48.601034+00:00
2021-06-06 23:08:31,796 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ae9ae037-6aa8-4cdf-b9c4-ecd1c7ada14e, queue=awx_private_queue)
[2021-06-06 23:08:31,796: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ae9ae037-6aa8-4cdf-b9c4-ecd1c7ada14e, queue=awx_private_queue)
2021-06-06 23:08:32,613 DEBUG awx.main.dispatch task 41279e8a-173c-49bf-bc57-e149c4dac89e is finished
2021-06-06 23:08:35,231 DEBUG awx.main.dispatch task 69cbf4c5-3208-4241-baf4-51b68abe4a2c is finished
2021-06-06 23:08:36,010 DEBUG awx.main.dispatch delivered d95d89f2-0dc2-4e81-9ac6-0c21655aaa0a to worker[182] qsize 0
2021-06-06 23:08:36,012 DEBUG awx.main.dispatch delivered 4758983f-227c-4099-a09e-5c83835a44ca to worker[196] qsize 0
2021-06-06 23:08:36,014 DEBUG awx.main.dispatch delivered 8db2d90c-4d83-4d34-9855-b76c33194441 to worker[193] qsize 0
2021-06-06 23:08:36,014 DEBUG awx.main.dispatch task 4758983f-227c-4099-a09e-5c83835a44ca starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:08:36,013 DEBUG awx.main.dispatch task d95d89f2-0dc2-4e81-9ac6-0c21655aaa0a starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:08:36,018 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:08:36,017 DEBUG awx.main.dispatch task 8db2d90c-4d83-4d34-9855-b76c33194441 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:08:36,019 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:08:36,022 WARNING awx.main.dispatch scaling up worker pid:204
2021-06-06 23:08:36,026 DEBUG awx.main.dispatch delivered 8e890b04-0477-493b-bb4a-869d99dc5573 to worker[204] qsize 0
2021-06-06 23:08:36,030 DEBUG awx.main.dispatch task 8e890b04-0477-493b-bb4a-869d99dc5573 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:08:36,036 WARNING awx.main.dispatch scaling up worker pid:206
2021-06-06 23:08:36,041 DEBUG awx.main.dispatch delivered ae9ae037-6aa8-4cdf-b9c4-ecd1c7ada14e to worker[206] qsize 0
2021-06-06 23:08:36,045 DEBUG awx.main.dispatch task ae9ae037-6aa8-4cdf-b9c4-ecd1c7ada14e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:08:36,047 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:08:36,114 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-06-06 23:08:36,114 DEBUG awx.main.dispatch task 8e890b04-0477-493b-bb4a-869d99dc5573 is finished
2021-06-06 23:08:36,129 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-06-06 23:08:36,129 DEBUG awx.main.dispatch task 8db2d90c-4d83-4d34-9855-b76c33194441 is finished
2021-06-06 23:08:38,617 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-06-06 23:08:38,617 DEBUG awx.main.dispatch task ae9ae037-6aa8-4cdf-b9c4-ecd1c7ada14e is finished
2021-06-06 23:08:44,583 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9249376e-f6bd-4b0b-ab42-4ecd82cfc66b, queue=awx_private_queue)
[2021-06-06 23:08:44,583: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9249376e-f6bd-4b0b-ab42-4ecd82cfc66b, queue=awx_private_queue)
2021-06-06 23:08:51,796 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bc4966de-49c1-4c9c-9210-c0e45ce71ec7, queue=awx_private_queue)
[2021-06-06 23:08:51,796: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bc4966de-49c1-4c9c-9210-c0e45ce71ec7, queue=awx_private_queue)
2021-06-06 23:08:52,308 DEBUG awx.main.dispatch task d95d89f2-0dc2-4e81-9ac6-0c21655aaa0a is finished
2021-06-06 23:08:48,473 DEBUG awx.main.dispatch task 4758983f-227c-4099-a09e-5c83835a44ca is finished
2021-06-06 23:08:56,808 DEBUG awx.main.dispatch task 8d01bfb7-3cb2-49a9-8019-62adc8abf4be is finished
2021-06-06 23:08:53,289 DEBUG awx.main.dispatch task 26067bdb-e8fd-44bb-9aae-231df4c73f2f is finished
RESULT 2
OKREADY
2021-06-06 23:09:11,523 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(aab1ecd9-cda8-4f7a-aea0-741453712879, queue=dev-awx-0)
[2021-06-06 23:09:11,523: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(aab1ecd9-cda8-4f7a-aea0-741453712879, queue=dev-awx-0)
2021-06-06 23:09:11,634 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3d78ab38-f48c-4bb3-ad5e-b8d8fd68031c, queue=dev-awx-0)
[2021-06-06 23:09:11,634: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3d78ab38-f48c-4bb3-ad5e-b8d8fd68031c, queue=dev-awx-0)
2021-06-06 23:09:11,796 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a3a47e74-6ff7-4917-b2e6-75e3772f8923, queue=awx_private_queue)
[2021-06-06 23:09:11,796: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a3a47e74-6ff7-4917-b2e6-75e3772f8923, queue=awx_private_queue)
2021-06-06 23:09:14,583 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e752e6a8-d493-40ca-96c7-209134b73fa2, queue=awx_private_queue)
[2021-06-06 23:09:14,583: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e752e6a8-d493-40ca-96c7-209134b73fa2, queue=awx_private_queue)
2021-06-06 23:09:31,797 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4645b320-fd83-4b14-b7bc-d45e213ee8cc, queue=awx_private_queue)
[2021-06-06 23:09:31,797: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4645b320-fd83-4b14-b7bc-d45e213ee8cc, queue=awx_private_queue)
2021-06-06 23:09:44,583 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bdfc3e79-796e-41a5-96e9-00480d9f9f1e, queue=awx_private_queue)
[2021-06-06 23:09:44,583: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bdfc3e79-796e-41a5-96e9-00480d9f9f1e, queue=awx_private_queue)
2021-06-06 23:09:51,797 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ad75d46d-4421-487c-8c76-caeac05d5bed, queue=awx_private_queue)
[2021-06-06 23:09:51,797: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ad75d46d-4421-487c-8c76-caeac05d5bed, queue=awx_private_queue)
RESULT 2
OKREADY
2021-06-06 23:08:48,047 DEBUG awx.main.dispatch delivered 9249376e-f6bd-4b0b-ab42-4ecd82cfc66b to worker[206] qsize 0
2021-06-06 23:10:46,896 DEBUG awx.main.dispatch delivered bc4966de-49c1-4c9c-9210-c0e45ce71ec7 to worker[193] qsize 0
2021-06-06 23:10:46,899 WARNING awx.main.dispatch scaling down worker pid:182
2021-06-06 23:10:46,900 WARNING awx.main.dispatch scaling down worker pid:196
2021-06-06 23:10:46,902 WARNING awx.main.dispatch scaling down worker pid:198
2021-06-06 23:10:46,900 DEBUG awx.main.dispatch task bc4966de-49c1-4c9c-9210-c0e45ce71ec7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:10:46,905 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:10:46,897 DEBUG awx.main.dispatch task 9249376e-f6bd-4b0b-ab42-4ecd82cfc66b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:10:46,903 WARNING awx.main.dispatch worker exiting gracefully pid:196
2021-06-06 23:10:46,902 WARNING awx.main.dispatch worker exiting gracefully pid:182
2021-06-06 23:10:46,906 WARNING awx.main.dispatch worker exiting gracefully pid:198
2021-06-06 23:10:11,540 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(dcfa963e-273b-463b-abb0-2412c21acec6, queue=dev-awx-0)
[2021-06-06 23:10:11,540: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(dcfa963e-273b-463b-abb0-2412c21acec6, queue=dev-awx-0)
2021-06-06 23:10:53,143 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(7ce458ac-0b76-4490-98d5-a54bb609f93f, queue=dev-awx-0)
[2021-06-06 23:10:53,143: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(7ce458ac-0b76-4490-98d5-a54bb609f93f, queue=dev-awx-0)
2021-06-06 23:10:53,159 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3d80903b-cafd-462c-9380-a670b4e5b06f, queue=awx_private_queue)
[2021-06-06 23:10:53,159: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3d80903b-cafd-462c-9380-a670b4e5b06f, queue=awx_private_queue)
2021-06-06 23:10:53,172 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(32e6e1fd-4f4f-4f4d-9727-18d434e376c0, queue=awx_private_queue)
[2021-06-06 23:10:53,172: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(32e6e1fd-4f4f-4f4d-9727-18d434e376c0, queue=awx_private_queue)
RESULT 2
OKREADY
2021-06-06 23:11:13,167 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0299e349-b7c7-4979-aa3b-f1f84108fa95, queue=awx_private_queue)
[2021-06-06 23:11:13,167: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0299e349-b7c7-4979-aa3b-f1f84108fa95, queue=awx_private_queue)
2021-06-06 23:11:23,172 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5ad6e0ba-de6b-47bf-98cd-26b747d3fcdb, queue=awx_private_queue)
[2021-06-06 23:11:23,172: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5ad6e0ba-de6b-47bf-98cd-26b747d3fcdb, queue=awx_private_queue)
2021-06-06 23:11:33,168 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b96fd114-aea8-4f21-a3d8-dddfefa43998, queue=awx_private_queue)
[2021-06-06 23:11:33,168: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b96fd114-aea8-4f21-a3d8-dddfefa43998, queue=awx_private_queue)
2021-06-06 23:11:33,182 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(cf82ade5-9b80-4b71-b38a-f42fb4b192d6, queue=dev-awx-0)
[2021-06-06 23:11:33,182: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(cf82ade5-9b80-4b71-b38a-f42fb4b192d6, queue=dev-awx-0)
2021-06-06 23:11:33,561 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:11:37,594 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:11:37,597 DEBUG awx.main.dispatch delivered aab1ecd9-cda8-4f7a-aea0-741453712879 to worker[204] qsize 0
2021-06-06 23:11:37,601 DEBUG awx.main.dispatch delivered 3d78ab38-f48c-4bb3-ad5e-b8d8fd68031c to worker[200] qsize 0
2021-06-06 23:11:37,602 DEBUG awx.main.dispatch task aab1ecd9-cda8-4f7a-aea0-741453712879 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:11:37,610 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:11:37,605 DEBUG awx.main.dispatch task 3d78ab38-f48c-4bb3-ad5e-b8d8fd68031c starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:11:37,614 WARNING awx.main.dispatch scaling up worker pid:211
2021-06-06 23:11:37,619 DEBUG awx.main.dispatch delivered a3a47e74-6ff7-4917-b2e6-75e3772f8923 to worker[211] qsize 0
2021-06-06 23:11:37,624 DEBUG awx.main.dispatch task a3a47e74-6ff7-4917-b2e6-75e3772f8923 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:11:37,626 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:11:37,631 WARNING awx.main.dispatch scaling up worker pid:213
2021-06-06 23:11:37,636 DEBUG awx.main.dispatch delivered e752e6a8-d493-40ca-96c7-209134b73fa2 to worker[213] qsize 0
2021-06-06 23:11:37,640 DEBUG awx.main.dispatch task e752e6a8-d493-40ca-96c7-209134b73fa2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:11:37,646 WARNING awx.main.dispatch scaling up worker pid:215
2021-06-06 23:11:37,653 DEBUG awx.main.dispatch delivered 4645b320-fd83-4b14-b7bc-d45e213ee8cc to worker[215] qsize 0
2021-06-06 23:11:37,658 DEBUG awx.main.dispatch task 4645b320-fd83-4b14-b7bc-d45e213ee8cc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:11:37,661 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:11:37,662 WARNING awx.main.dispatch scaling up worker pid:217
2021-06-06 23:11:37,666 DEBUG awx.main.dispatch delivered bdfc3e79-796e-41a5-96e9-00480d9f9f1e to worker[217] qsize 0
2021-06-06 23:11:37,669 DEBUG awx.main.dispatch task bdfc3e79-796e-41a5-96e9-00480d9f9f1e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:11:37,675 WARNING awx.main.dispatch scaling up worker pid:219
2021-06-06 23:11:37,679 DEBUG awx.main.dispatch delivered ad75d46d-4421-487c-8c76-caeac05d5bed to worker[219] qsize 0
2021-06-06 23:11:37,683 DEBUG awx.main.dispatch task ad75d46d-4421-487c-8c76-caeac05d5bed starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:11:37,686 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:11:37,882 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:08:21.541318+00:00
2021-06-06 23:11:38,330 DEBUG awx.main.dispatch task 3d78ab38-f48c-4bb3-ad5e-b8d8fd68031c is finished
2021-06-06 23:11:40,081 DEBUG awx.main.dispatch task bc4966de-49c1-4c9c-9210-c0e45ce71ec7 is finished
2021-06-06 23:11:41,043 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-06-06 23:11:41,043 DEBUG awx.main.dispatch task e752e6a8-d493-40ca-96c7-209134b73fa2 is finished
2021-06-06 23:11:41,129 DEBUG awx.main.dispatch task 9249376e-f6bd-4b0b-ab42-4ecd82cfc66b is finished
2021-06-06 23:11:41,270 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:11:41,283 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:11:41,289 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:11:33.562068+00:00
2021-06-06 23:11:41,292 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-06-06 23:11:41,293 DEBUG awx.main.dispatch task ad75d46d-4421-487c-8c76-caeac05d5bed is finished
2021-06-06 23:11:41,307 DEBUG awx.main.dispatch task bdfc3e79-796e-41a5-96e9-00480d9f9f1e is finished
2021-06-06 23:11:42,060 DEBUG awx.main.dispatch delivered dcfa963e-273b-463b-abb0-2412c21acec6 to worker[217] qsize 0
2021-06-06 23:11:42,062 DEBUG awx.main.dispatch delivered 7ce458ac-0b76-4490-98d5-a54bb609f93f to worker[206] qsize 0
2021-06-06 23:11:42,062 DEBUG awx.main.dispatch task dcfa963e-273b-463b-abb0-2412c21acec6 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:11:42,064 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:11:42,065 DEBUG awx.main.dispatch task 7ce458ac-0b76-4490-98d5-a54bb609f93f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:11:42,066 DEBUG awx.main.dispatch delivered 3d80903b-cafd-462c-9380-a670b4e5b06f to worker[200] qsize 0
2021-06-06 23:11:42,068 DEBUG awx.main.dispatch delivered 32e6e1fd-4f4f-4f4d-9727-18d434e376c0 to worker[213] qsize 0
2021-06-06 23:11:42,070 DEBUG awx.main.dispatch delivered 0299e349-b7c7-4979-aa3b-f1f84108fa95 to worker[193] qsize 0
2021-06-06 23:11:42,070 DEBUG awx.main.dispatch task 3d80903b-cafd-462c-9380-a670b4e5b06f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:11:42,071 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:11:42,073 DEBUG awx.main.dispatch delivered 5ad6e0ba-de6b-47bf-98cd-26b747d3fcdb to worker[219] qsize 0
2021-06-06 23:11:42,070 DEBUG awx.main.dispatch task 32e6e1fd-4f4f-4f4d-9727-18d434e376c0 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:11:42,073 DEBUG awx.main.dispatch task 0299e349-b7c7-4979-aa3b-f1f84108fa95 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:11:42,074 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:11:42,076 DEBUG awx.main.dispatch task 5ad6e0ba-de6b-47bf-98cd-26b747d3fcdb starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:11:42,082 WARNING awx.main.dispatch scaling up worker pid:224
2021-06-06 23:11:42,087 DEBUG awx.main.dispatch delivered b96fd114-aea8-4f21-a3d8-dddfefa43998 to worker[224] qsize 0
2021-06-06 23:11:42,093 DEBUG awx.main.dispatch task b96fd114-aea8-4f21-a3d8-dddfefa43998 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:11:42,096 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:11:42,400 DEBUG awx.main.dispatch task a3a47e74-6ff7-4917-b2e6-75e3772f8923 is finished
2021-06-06 23:11:58,293 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:11:58,323 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:11:53,151 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(166a79f0-b855-4b3a-a33d-219f0fa8ef28, queue=dev-awx-0)
[2021-06-06 23:11:53,151: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(166a79f0-b855-4b3a-a33d-219f0fa8ef28, queue=dev-awx-0)
2021-06-06 23:11:58,980 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bea0f13c-2b1a-4ecd-be69-b1fdad1c70eb, queue=awx_private_queue)
[2021-06-06 23:11:58,980: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bea0f13c-2b1a-4ecd-be69-b1fdad1c70eb, queue=awx_private_queue)
2021-06-06 23:11:58,996 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5ac92fdd-3a36-4451-b8e0-5fcd5d3201fb, queue=awx_private_queue)
[2021-06-06 23:11:58,996: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5ac92fdd-3a36-4451-b8e0-5fcd5d3201fb, queue=awx_private_queue)
2021-06-06 23:11:59,210 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-06-06 23:11:59,210 DEBUG awx.main.dispatch task 32e6e1fd-4f4f-4f4d-9727-18d434e376c0 is finished
2021-06-06 23:11:59,268 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:11:41.283628+00:00
2021-06-06 23:12:00,149 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-06-06 23:12:00,150 DEBUG awx.main.dispatch task b96fd114-aea8-4f21-a3d8-dddfefa43998 is finished
RESULT 2
OKREADY
2021-06-06 23:12:00,356 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-06-06 23:12:00,356 DEBUG awx.main.dispatch task 3d80903b-cafd-462c-9380-a670b4e5b06f is finished
2021-06-06 23:12:00,357 DEBUG awx.main.dispatch task 7ce458ac-0b76-4490-98d5-a54bb609f93f is finished
2021-06-06 23:12:01,249 WARNING awx.main.tasks Rejoining the cluster as instance dev-awx-0.
2021-06-06 23:12:01,249 DEBUG awx.main.dispatch task dcfa963e-273b-463b-abb0-2412c21acec6 is finished
2021-06-06 23:12:13,897 DEBUG awx.main.dispatch delivered cf82ade5-9b80-4b71-b38a-f42fb4b192d6 to worker[213] qsize 0
2021-06-06 23:12:13,904 DEBUG awx.main.dispatch delivered 166a79f0-b855-4b3a-a33d-219f0fa8ef28 to worker[211] qsize 0
2021-06-06 23:12:13,903 DEBUG awx.main.dispatch task cf82ade5-9b80-4b71-b38a-f42fb4b192d6 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:12:13,905 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:12:13,906 DEBUG awx.main.dispatch delivered bea0f13c-2b1a-4ecd-be69-b1fdad1c70eb to worker[224] qsize 0
2021-06-06 23:12:13,906 DEBUG awx.main.dispatch task 166a79f0-b855-4b3a-a33d-219f0fa8ef28 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:12:13,908 DEBUG awx.main.dispatch delivered 5ac92fdd-3a36-4451-b8e0-5fcd5d3201fb to worker[206] qsize 0
2021-06-06 23:12:13,908 DEBUG awx.main.dispatch task bea0f13c-2b1a-4ecd-be69-b1fdad1c70eb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:12:13,911 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:12:13,911 DEBUG awx.main.dispatch task 5ac92fdd-3a36-4451-b8e0-5fcd5d3201fb starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:12:16,688 DEBUG awx.main.dispatch task 4645b320-fd83-4b14-b7bc-d45e213ee8cc is finished
2021-06-06 23:12:18,986 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c74c9e06-ef0a-4312-a373-c19a02788c27, queue=awx_private_queue)
[2021-06-06 23:12:18,986: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c74c9e06-ef0a-4312-a373-c19a02788c27, queue=awx_private_queue)
2021-06-06 23:12:19,010 DEBUG awx.main.dispatch delivered c74c9e06-ef0a-4312-a373-c19a02788c27 to worker[215] qsize 0
2021-06-06 23:12:19,013 DEBUG awx.main.dispatch task c74c9e06-ef0a-4312-a373-c19a02788c27 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:12:19,015 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:12:27,262 DEBUG awx.main.dispatch task 5ad6e0ba-de6b-47bf-98cd-26b747d3fcdb is finished
2021-06-06 23:12:28,996 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(661c2c5e-56b2-4c3e-aba2-b8d96ffd8a2f, queue=awx_private_queue)
[2021-06-06 23:12:28,996: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(661c2c5e-56b2-4c3e-aba2-b8d96ffd8a2f, queue=awx_private_queue)
2021-06-06 23:12:29,027 DEBUG awx.main.dispatch delivered 661c2c5e-56b2-4c3e-aba2-b8d96ffd8a2f to worker[219] qsize 0
2021-06-06 23:12:29,031 DEBUG awx.main.dispatch task 661c2c5e-56b2-4c3e-aba2-b8d96ffd8a2f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:12:33,183 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(74259c3e-01a6-40a6-8fc0-ef1e0260d1f2, queue=dev-awx-0)
[2021-06-06 23:12:33,183: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(74259c3e-01a6-40a6-8fc0-ef1e0260d1f2, queue=dev-awx-0)
2021-06-06 23:12:33,205 WARNING awx.main.dispatch scaling down worker pid:200
2021-06-06 23:12:33,207 WARNING awx.main.dispatch scaling down worker pid:217
2021-06-06 23:12:33,207 WARNING awx.main.dispatch worker exiting gracefully pid:200
2021-06-06 23:12:33,210 WARNING awx.main.dispatch worker exiting gracefully pid:217
2021-06-06 23:12:38,981 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(684e1cb1-684b-48cb-ab13-e24b905c0efb, queue=awx_private_queue)
[2021-06-06 23:12:38,981: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(684e1cb1-684b-48cb-ab13-e24b905c0efb, queue=awx_private_queue)
2021-06-06 23:12:53,151 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(43601180-be5a-4cb7-82d7-396f45b2005d, queue=dev-awx-0)
[2021-06-06 23:12:53,151: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(43601180-be5a-4cb7-82d7-396f45b2005d, queue=dev-awx-0)
2021-06-06 23:12:55,805 DEBUG awx.main.dispatch task aab1ecd9-cda8-4f7a-aea0-741453712879 is finished
RESULT 2
OKREADY
2021-06-06 23:13:18,073 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 23:12:58,981 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(439166ef-09a3-40f7-84d8-dff8a1f8b752, queue=awx_private_queue)
[2021-06-06 23:12:58,981: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(439166ef-09a3-40f7-84d8-dff8a1f8b752, queue=awx_private_queue)
2021-06-06 23:13:18,737 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2868d3b2-217c-4ff9-875b-e4b5e93e49f5, queue=awx_private_queue)
[2021-06-06 23:13:18,737: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2868d3b2-217c-4ff9-875b-e4b5e93e49f5, queue=awx_private_queue)
2021-06-06 23:13:19,853 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:13:30,964 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-06-06 23:13:33,187 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a6b6badf-1440-48ad-8e78-6bb5c2c030ca, queue=dev-awx-0)
[2021-06-06 23:13:33,187: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a6b6badf-1440-48ad-8e78-6bb5c2c030ca, queue=dev-awx-0)
2021-06-06 23:13:33,203 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1bd090b6-9740-43cf-9b59-89a6a62d0b83, queue=awx_private_queue)
[2021-06-06 23:13:33,203: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1bd090b6-9740-43cf-9b59-89a6a62d0b83, queue=awx_private_queue)
2021-06-06 23:13:48,740 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0234430f-1b1e-4655-9f85-5b9c4394ab65, queue=awx_private_queue)
[2021-06-06 23:13:48,740: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0234430f-1b1e-4655-9f85-5b9c4394ab65, queue=awx_private_queue)
2021-06-06 23:13:53,154 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(550ccda0-a03a-4b2d-b071-de3f840f8982, queue=dev-awx-0)
[2021-06-06 23:13:53,154: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(550ccda0-a03a-4b2d-b071-de3f840f8982, queue=dev-awx-0)
2021-06-06 23:13:53,204 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f0a24c5d-02b4-4675-aa44-72e97fb57e5d, queue=awx_private_queue)
[2021-06-06 23:13:53,204: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f0a24c5d-02b4-4675-aa44-72e97fb57e5d, queue=awx_private_queue)
2021-06-06 23:13:56,188 DEBUG awx.main.dispatch task 166a79f0-b855-4b3a-a33d-219f0fa8ef28 is finished
2021-06-06 23:13:30,965 DEBUG awx.main.dispatch task 0299e349-b7c7-4979-aa3b-f1f84108fa95 is finished
RESULT 2
OKREADY
2021-06-06 23:14:05,306 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 23:11:58.294348+00:00
2021-06-06 23:14:05,796 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-06-06 23:14:05,797 DEBUG awx.main.dispatch task c74c9e06-ef0a-4312-a373-c19a02788c27 is finished
2021-06-06 23:14:13,222 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(71eca0ab-3a5b-4b17-8177-06e3a299ab21, queue=awx_private_queue)
[2021-06-06 23:14:13,222: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(71eca0ab-3a5b-4b17-8177-06e3a299ab21, queue=awx_private_queue)
2021-06-06 23:14:17,678 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-06-06 23:14:17,679 DEBUG awx.main.dispatch task 661c2c5e-56b2-4c3e-aba2-b8d96ffd8a2f is finished
2021-06-06 23:14:18,741 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(57177498-d779-4c82-a772-17ee52356eb2, queue=awx_private_queue)
[2021-06-06 23:14:18,741: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(57177498-d779-4c82-a772-17ee52356eb2, queue=awx_private_queue)
2021-06-06 23:14:20,740 DEBUG awx.main.dispatch delivered 74259c3e-01a6-40a6-8fc0-ef1e0260d1f2 to worker[219] qsize 0
2021-06-06 23:14:20,747 DEBUG awx.main.dispatch task 74259c3e-01a6-40a6-8fc0-ef1e0260d1f2 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:14:20,749 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:14:20,753 DEBUG awx.main.dispatch delivered 684e1cb1-684b-48cb-ab13-e24b905c0efb to worker[215] qsize 0
2021-06-06 23:14:20,761 DEBUG awx.main.dispatch task 684e1cb1-684b-48cb-ab13-e24b905c0efb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:14:20,764 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:14:20,768 DEBUG awx.main.dispatch delivered 43601180-be5a-4cb7-82d7-396f45b2005d to worker[193] qsize 0
2021-06-06 23:14:20,792 DEBUG awx.main.dispatch delivered 439166ef-09a3-40f7-84d8-dff8a1f8b752 to worker[204] qsize 0
2021-06-06 23:14:20,795 DEBUG awx.main.dispatch delivered 2868d3b2-217c-4ff9-875b-e4b5e93e49f5 to worker[211] qsize 0
2021-06-06 23:14:20,793 DEBUG awx.main.dispatch task 43601180-be5a-4cb7-82d7-396f45b2005d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:14:20,799 DEBUG awx.main.dispatch task 2868d3b2-217c-4ff9-875b-e4b5e93e49f5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:14:20,775 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-06-06 23:14:20,806 DEBUG awx.main.dispatch task 74259c3e-01a6-40a6-8fc0-ef1e0260d1f2 is finished
2021-06-06 23:14:20,776 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:14:20,815 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:14:20,814 DEBUG awx.main.dispatch task 684e1cb1-684b-48cb-ab13-e24b905c0efb is finished
2021-06-06 23:14:20,805 DEBUG awx.main.dispatch task 439166ef-09a3-40f7-84d8-dff8a1f8b752 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:14:20,817 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:14:20,823 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139824236545600_1" does not exist
2021-06-06 23:14:20,822 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:14:20,831 DEBUG awx.main.dispatch task 43601180-be5a-4cb7-82d7-396f45b2005d is finished
2021-06-06 23:14:20,831 DEBUG awx.main.dispatch task 2868d3b2-217c-4ff9-875b-e4b5e93e49f5 is finished
2021-06-06 23:14:20,833 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:14:20,843 DEBUG awx.main.dispatch task 439166ef-09a3-40f7-84d8-dff8a1f8b752 is finished
2021-06-06 23:14:21,472 DEBUG awx.main.dispatch task 5ac92fdd-3a36-4451-b8e0-5fcd5d3201fb is finished
2021-06-06 23:14:28,067 DEBUG awx.main.dispatch task cf82ade5-9b80-4b71-b38a-f42fb4b192d6 is finished
2021-06-06 23:14:33,188 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(81028dfa-ad25-4c80-8769-ffe0af3c6fc4, queue=dev-awx-0)
[2021-06-06 23:14:33,188: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(81028dfa-ad25-4c80-8769-ffe0af3c6fc4, queue=dev-awx-0)
2021-06-06 23:14:33,214 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ce194c9a-b419-4880-ab76-e4999a7818a4, queue=awx_private_queue)
[2021-06-06 23:14:33,214: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ce194c9a-b419-4880-ab76-e4999a7818a4, queue=awx_private_queue)
2021-06-06 23:14:48,761 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d79d36f6-0d94-47bd-a45f-b521c5803289, queue=awx_private_queue)
[2021-06-06 23:14:48,761: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d79d36f6-0d94-47bd-a45f-b521c5803289, queue=awx_private_queue)
2021-06-06 23:14:53,152 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(85ce9c74-d427-4099-ab51-f637989405d4, queue=dev-awx-0)
[2021-06-06 23:14:53,152: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(85ce9c74-d427-4099-ab51-f637989405d4, queue=dev-awx-0)
2021-06-06 23:14:53,217 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5362a259-264f-438d-a81a-247c405ffe9b, queue=awx_private_queue)
[2021-06-06 23:14:53,217: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5362a259-264f-438d-a81a-247c405ffe9b, queue=awx_private_queue)
RESULT 2
OKREADY
2021-06-06 23:15:02,081 DEBUG awx.main.dispatch delivered a6b6badf-1440-48ad-8e78-6bb5c2c030ca to worker[206] qsize 0
2021-06-06 23:15:02,106 DEBUG awx.main.dispatch delivered 1bd090b6-9740-43cf-9b59-89a6a62d0b83 to worker[193] qsize 0
2021-06-06 23:15:02,108 DEBUG awx.main.dispatch delivered 0234430f-1b1e-4655-9f85-5b9c4394ab65 to worker[219] qsize 0
2021-06-06 23:15:02,109 DEBUG awx.main.dispatch delivered 550ccda0-a03a-4b2d-b071-de3f840f8982 to worker[211] qsize 0
2021-06-06 23:15:02,107 DEBUG awx.main.dispatch task a6b6badf-1440-48ad-8e78-6bb5c2c030ca starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:15:02,112 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:15:02,111 DEBUG awx.main.dispatch delivered f0a24c5d-02b4-4675-aa44-72e97fb57e5d to worker[215] qsize 0
2021-06-06 23:15:02,113 DEBUG awx.main.dispatch delivered 71eca0ab-3a5b-4b17-8177-06e3a299ab21 to worker[213] qsize 0
2021-06-06 23:15:02,116 DEBUG awx.main.dispatch delivered 57177498-d779-4c82-a772-17ee52356eb2 to worker[204] qsize 0
2021-06-06 23:15:02,118 DEBUG awx.main.dispatch task f0a24c5d-02b4-4675-aa44-72e97fb57e5d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:15:02,121 DEBUG awx.main.dispatch task 0234430f-1b1e-4655-9f85-5b9c4394ab65 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:15:02,129 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:15:02,125 DEBUG awx.main.dispatch task 71eca0ab-3a5b-4b17-8177-06e3a299ab21 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:15:02,135 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:15:02,134 DEBUG awx.main.dispatch task 57177498-d779-4c82-a772-17ee52356eb2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:15:02,146 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:15:02,139 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-06-06 23:15:02,145 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:15:02,148 DEBUG awx.main.dispatch task a6b6badf-1440-48ad-8e78-6bb5c2c030ca is finished
2021-06-06 23:15:02,148 DEBUG awx.main.dispatch task f0a24c5d-02b4-4675-aa44-72e97fb57e5d is finished
2021-06-06 23:15:02,150 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-06-06 23:15:02,150 DEBUG awx.main.dispatch task 57177498-d779-4c82-a772-17ee52356eb2 is finished
2021-06-06 23:15:02,156 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:15:02,156 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:15:02,161 DEBUG awx.main.dispatch task 71eca0ab-3a5b-4b17-8177-06e3a299ab21 is finished
2021-06-06 23:15:02,161 DEBUG awx.main.dispatch task 0234430f-1b1e-4655-9f85-5b9c4394ab65 is finished
2021-06-06 23:15:02,123 DEBUG awx.main.dispatch task 550ccda0-a03a-4b2d-b071-de3f840f8982 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:15:02,121 DEBUG awx.main.dispatch task 1bd090b6-9740-43cf-9b59-89a6a62d0b83 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:15:02,200 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:15:02,202 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139824236545600_2" does not exist
2021-06-06 23:15:02,204 DEBUG awx.main.dispatch task 550ccda0-a03a-4b2d-b071-de3f840f8982 is finished
2021-06-06 23:15:02,202 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:15:02,207 DEBUG awx.main.dispatch task 1bd090b6-9740-43cf-9b59-89a6a62d0b83 is finished
2021-06-06 23:15:02,332 DEBUG awx.main.dispatch delivered 81028dfa-ad25-4c80-8769-ffe0af3c6fc4 to worker[193] qsize 0
2021-06-06 23:15:02,335 DEBUG awx.main.dispatch delivered ce194c9a-b419-4880-ab76-e4999a7818a4 to worker[213] qsize 0
2021-06-06 23:15:02,336 DEBUG awx.main.dispatch task 81028dfa-ad25-4c80-8769-ffe0af3c6fc4 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:15:02,338 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:15:02,338 DEBUG awx.main.dispatch task ce194c9a-b419-4880-ab76-e4999a7818a4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:15:02,341 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:15:02,339 DEBUG awx.main.dispatch delivered d79d36f6-0d94-47bd-a45f-b521c5803289 to worker[206] qsize 0
2021-06-06 23:15:02,345 DEBUG awx.main.dispatch delivered 85ce9c74-d427-4099-ab51-f637989405d4 to worker[215] qsize 0
2021-06-06 23:15:02,345 DEBUG awx.main.dispatch task d79d36f6-0d94-47bd-a45f-b521c5803289 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:15:02,347 DEBUG awx.main.dispatch delivered 5362a259-264f-438d-a81a-247c405ffe9b to worker[211] qsize 0
2021-06-06 23:15:02,348 DEBUG awx.main.dispatch task 85ce9c74-d427-4099-ab51-f637989405d4 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:15:02,351 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-06-06 23:15:02,353 DEBUG awx.main.dispatch task 81028dfa-ad25-4c80-8769-ffe0af3c6fc4 is finished
2021-06-06 23:15:02,352 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:15:02,354 DEBUG awx.main.dispatch task 5362a259-264f-438d-a81a-247c405ffe9b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:15:02,354 DEBUG awx.main.dispatch task ce194c9a-b419-4880-ab76-e4999a7818a4 is finished
2021-06-06 23:15:02,356 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:15:02,356 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:15:02,359 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:15:02,362 DEBUG awx.main.dispatch task d79d36f6-0d94-47bd-a45f-b521c5803289 is finished
2021-06-06 23:15:02,363 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139824236545600_1" does not exist
2021-06-06 23:15:02,365 DEBUG awx.main.dispatch task 85ce9c74-d427-4099-ab51-f637989405d4 is finished
2021-06-06 23:15:02,365 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:15:02,367 DEBUG awx.main.dispatch task 5362a259-264f-438d-a81a-247c405ffe9b is finished
2021-06-06 23:15:03,192 DEBUG awx.main.dispatch task bea0f13c-2b1a-4ecd-be69-b1fdad1c70eb is finished
2021-06-06 23:15:13,225 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(df5abd0b-24f7-4b19-afcf-a220e5d721a7, queue=awx_private_queue)
[2021-06-06 23:15:13,225: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(df5abd0b-24f7-4b19-afcf-a220e5d721a7, queue=awx_private_queue)
2021-06-06 23:15:13,244 DEBUG awx.main.dispatch delivered df5abd0b-24f7-4b19-afcf-a220e5d721a7 to worker[211] qsize 0
2021-06-06 23:15:13,276 DEBUG awx.main.dispatch task df5abd0b-24f7-4b19-afcf-a220e5d721a7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:15:13,277 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:15:13,287 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:15:13,289 DEBUG awx.main.dispatch task df5abd0b-24f7-4b19-afcf-a220e5d721a7 is finished
2021-06-06 23:15:18,764 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dff5792b-c3e4-4b99-bfd8-1f732bdb4fb9, queue=awx_private_queue)
[2021-06-06 23:15:18,764: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dff5792b-c3e4-4b99-bfd8-1f732bdb4fb9, queue=awx_private_queue)
2021-06-06 23:15:18,779 DEBUG awx.main.dispatch delivered dff5792b-c3e4-4b99-bfd8-1f732bdb4fb9 to worker[206] qsize 0
2021-06-06 23:15:18,783 DEBUG awx.main.dispatch task dff5792b-c3e4-4b99-bfd8-1f732bdb4fb9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:15:18,794 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:15:18,797 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:15:18,798 DEBUG awx.main.dispatch task dff5792b-c3e4-4b99-bfd8-1f732bdb4fb9 is finished
2021-06-06 23:15:33,194 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(ecc6ec6b-e321-4585-b9e8-517c09b3b0d8, queue=dev-awx-0)
[2021-06-06 23:15:33,194: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(ecc6ec6b-e321-4585-b9e8-517c09b3b0d8, queue=dev-awx-0)
2021-06-06 23:15:33,214 WARNING awx.main.dispatch scaling down worker pid:193
2021-06-06 23:15:33,218 WARNING awx.main.dispatch scaling down worker pid:204
2021-06-06 23:15:33,221 WARNING awx.main.dispatch worker exiting gracefully pid:193
2021-06-06 23:15:33,221 WARNING awx.main.dispatch worker exiting gracefully pid:204
2021-06-06 23:15:33,225 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5819228a-7d25-4427-9816-9b601027ff52, queue=awx_private_queue)
[2021-06-06 23:15:33,225: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5819228a-7d25-4427-9816-9b601027ff52, queue=awx_private_queue)
2021-06-06 23:15:33,220 WARNING awx.main.dispatch scaling down worker pid:206
2021-06-06 23:15:33,235 WARNING awx.main.dispatch scaling down worker pid:211
2021-06-06 23:15:33,238 WARNING awx.main.dispatch worker exiting gracefully pid:206
2021-06-06 23:15:33,238 WARNING awx.main.dispatch worker exiting gracefully pid:211
2021-06-06 23:15:37,325 DEBUG awx.main.dispatch delivered ecc6ec6b-e321-4585-b9e8-517c09b3b0d8 to worker[213] qsize 0
2021-06-06 23:15:37,327 DEBUG awx.main.dispatch delivered 5819228a-7d25-4427-9816-9b601027ff52 to worker[219] qsize 0
2021-06-06 23:15:37,330 DEBUG awx.main.dispatch task ecc6ec6b-e321-4585-b9e8-517c09b3b0d8 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:15:37,334 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:15:37,331 DEBUG awx.main.dispatch task 5819228a-7d25-4427-9816-9b601027ff52 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:15:37,336 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:15:37,350 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:15:37,350 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-06-06 23:15:37,353 DEBUG awx.main.dispatch task 5819228a-7d25-4427-9816-9b601027ff52 is finished
2021-06-06 23:15:37,353 DEBUG awx.main.dispatch task ecc6ec6b-e321-4585-b9e8-517c09b3b0d8 is finished
2021-06-06 23:15:48,771 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dc529615-79ee-4afa-a7e6-17e88e406e7c, queue=awx_private_queue)
[2021-06-06 23:15:48,771: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dc529615-79ee-4afa-a7e6-17e88e406e7c, queue=awx_private_queue)
2021-06-06 23:15:48,788 DEBUG awx.main.dispatch delivered dc529615-79ee-4afa-a7e6-17e88e406e7c to worker[215] qsize 0
2021-06-06 23:15:48,794 DEBUG awx.main.dispatch task dc529615-79ee-4afa-a7e6-17e88e406e7c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:15:48,808 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:15:48,814 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:15:48,816 DEBUG awx.main.dispatch task dc529615-79ee-4afa-a7e6-17e88e406e7c is finished
2021-06-06 23:15:53,152 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(245a774b-8438-4792-a4ac-b4aa6bbbb9e2, queue=dev-awx-0)
[2021-06-06 23:15:53,152: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(245a774b-8438-4792-a4ac-b4aa6bbbb9e2, queue=dev-awx-0)
2021-06-06 23:15:53,172 DEBUG awx.main.dispatch delivered 245a774b-8438-4792-a4ac-b4aa6bbbb9e2 to worker[213] qsize 0
2021-06-06 23:15:53,177 DEBUG awx.main.dispatch task 245a774b-8438-4792-a4ac-b4aa6bbbb9e2 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:15:53,194 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139824236545600_1" does not exist
2021-06-06 23:15:53,196 DEBUG awx.main.dispatch task 245a774b-8438-4792-a4ac-b4aa6bbbb9e2 is finished
2021-06-06 23:15:53,225 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(aa8da5e0-4814-4ceb-9cec-e43df404f688, queue=awx_private_queue)
[2021-06-06 23:15:53,225: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(aa8da5e0-4814-4ceb-9cec-e43df404f688, queue=awx_private_queue)
2021-06-06 23:15:53,247 DEBUG awx.main.dispatch delivered aa8da5e0-4814-4ceb-9cec-e43df404f688 to worker[224] qsize 0
2021-06-06 23:15:53,251 DEBUG awx.main.dispatch task aa8da5e0-4814-4ceb-9cec-e43df404f688 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:15:53,255 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:15:53,271 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:15:53,280 DEBUG awx.main.dispatch task aa8da5e0-4814-4ceb-9cec-e43df404f688 is finished
RESULT 2
OKREADY
2021-06-06 23:16:13,227 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ad296101-ceff-4b58-b4a0-8b6b1fc2a67a, queue=awx_private_queue)
[2021-06-06 23:16:13,227: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ad296101-ceff-4b58-b4a0-8b6b1fc2a67a, queue=awx_private_queue)
2021-06-06 23:16:13,267 DEBUG awx.main.dispatch delivered ad296101-ceff-4b58-b4a0-8b6b1fc2a67a to worker[215] qsize 0
2021-06-06 23:16:13,278 DEBUG awx.main.dispatch task ad296101-ceff-4b58-b4a0-8b6b1fc2a67a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:16:13,280 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:16:13,290 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:16:13,295 DEBUG awx.main.dispatch task ad296101-ceff-4b58-b4a0-8b6b1fc2a67a is finished
2021-06-06 23:16:18,773 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(33293549-f601-4281-9620-00cfe209f63a, queue=awx_private_queue)
[2021-06-06 23:16:18,773: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(33293549-f601-4281-9620-00cfe209f63a, queue=awx_private_queue)
2021-06-06 23:16:18,789 DEBUG awx.main.dispatch delivered 33293549-f601-4281-9620-00cfe209f63a to worker[213] qsize 0
2021-06-06 23:16:18,792 DEBUG awx.main.dispatch task 33293549-f601-4281-9620-00cfe209f63a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:16:18,801 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:16:18,805 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:16:18,807 DEBUG awx.main.dispatch task 33293549-f601-4281-9620-00cfe209f63a is finished
2021-06-06 23:16:33,191 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(551ae3df-14f6-4cc2-8454-cda142c016bc, queue=dev-awx-0)
[2021-06-06 23:16:33,191: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(551ae3df-14f6-4cc2-8454-cda142c016bc, queue=dev-awx-0)
2021-06-06 23:16:33,227 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b58fafb5-7b30-4f59-8f1c-c15c5aafc867, queue=awx_private_queue)
[2021-06-06 23:16:33,227: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b58fafb5-7b30-4f59-8f1c-c15c5aafc867, queue=awx_private_queue)
2021-06-06 23:16:33,220 DEBUG awx.main.dispatch delivered 551ae3df-14f6-4cc2-8454-cda142c016bc to worker[219] qsize 0
2021-06-06 23:16:33,402 DEBUG awx.main.dispatch delivered b58fafb5-7b30-4f59-8f1c-c15c5aafc867 to worker[215] qsize 0
2021-06-06 23:16:33,404 DEBUG awx.main.dispatch task b58fafb5-7b30-4f59-8f1c-c15c5aafc867 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:16:33,406 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:16:33,405 DEBUG awx.main.dispatch task 551ae3df-14f6-4cc2-8454-cda142c016bc starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 23:16:33,408 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 23:16:33,416 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:16:33,418 DEBUG awx.main.dispatch task b58fafb5-7b30-4f59-8f1c-c15c5aafc867 is finished
2021-06-06 23:16:33,421 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-06-06 23:16:33,425 DEBUG awx.main.dispatch task 551ae3df-14f6-4cc2-8454-cda142c016bc is finished
2021-06-06 23:16:48,784 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4feb7f78-1cea-401f-a8e2-4ae619f51fe4, queue=awx_private_queue)
[2021-06-06 23:16:48,784: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4feb7f78-1cea-401f-a8e2-4ae619f51fe4, queue=awx_private_queue)
2021-06-06 23:16:48,807 DEBUG awx.main.dispatch delivered 4feb7f78-1cea-401f-a8e2-4ae619f51fe4 to worker[224] qsize 0
2021-06-06 23:16:48,814 DEBUG awx.main.dispatch task 4feb7f78-1cea-401f-a8e2-4ae619f51fe4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:16:48,826 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:16:48,831 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:16:48,834 DEBUG awx.main.dispatch task 4feb7f78-1cea-401f-a8e2-4ae619f51fe4 is finished
2021-06-06 23:16:53,152 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5aecfa74-2c54-43ba-a94d-dc6dbabb1f00, queue=dev-awx-0)
[2021-06-06 23:16:53,152: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5aecfa74-2c54-43ba-a94d-dc6dbabb1f00, queue=dev-awx-0)
2021-06-06 23:16:53,167 DEBUG awx.main.dispatch delivered 5aecfa74-2c54-43ba-a94d-dc6dbabb1f00 to worker[219] qsize 0
2021-06-06 23:16:53,173 DEBUG awx.main.dispatch task 5aecfa74-2c54-43ba-a94d-dc6dbabb1f00 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:16:53,189 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139824236545600_1" does not exist
2021-06-06 23:16:53,191 DEBUG awx.main.dispatch task 5aecfa74-2c54-43ba-a94d-dc6dbabb1f00 is finished
2021-06-06 23:16:53,227 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(50415311-b6e4-4543-adbd-d83fc4b56890, queue=awx_private_queue)
[2021-06-06 23:16:53,227: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(50415311-b6e4-4543-adbd-d83fc4b56890, queue=awx_private_queue)
2021-06-06 23:16:53,242 DEBUG awx.main.dispatch delivered 50415311-b6e4-4543-adbd-d83fc4b56890 to worker[224] qsize 0
2021-06-06 23:16:53,245 DEBUG awx.main.dispatch task 50415311-b6e4-4543-adbd-d83fc4b56890 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:16:53,247 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:16:53,268 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:16:53,270 DEBUG awx.main.dispatch task 50415311-b6e4-4543-adbd-d83fc4b56890 is finished
RESULT 2
OKREADY
2021-06-06 23:17:13,227 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4761f9e9-7947-48e8-8937-5b51b3d11570, queue=awx_private_queue)
[2021-06-06 23:17:13,227: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4761f9e9-7947-48e8-8937-5b51b3d11570, queue=awx_private_queue)
2021-06-06 23:17:13,245 DEBUG awx.main.dispatch delivered 4761f9e9-7947-48e8-8937-5b51b3d11570 to worker[213] qsize 0
2021-06-06 23:17:13,257 DEBUG awx.main.dispatch task 4761f9e9-7947-48e8-8937-5b51b3d11570 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:17:13,268 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:17:13,284 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:17:13,286 DEBUG awx.main.dispatch task 4761f9e9-7947-48e8-8937-5b51b3d11570 is finished
2021-06-06 23:17:18,776 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f5aee17e-9974-4e4c-8065-62ef472cd371, queue=awx_private_queue)
[2021-06-06 23:17:18,776: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f5aee17e-9974-4e4c-8065-62ef472cd371, queue=awx_private_queue)
2021-06-06 23:17:18,789 DEBUG awx.main.dispatch delivered f5aee17e-9974-4e4c-8065-62ef472cd371 to worker[219] qsize 0
2021-06-06 23:17:18,794 DEBUG awx.main.dispatch task f5aee17e-9974-4e4c-8065-62ef472cd371 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:17:18,808 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:17:18,814 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:17:18,816 DEBUG awx.main.dispatch task f5aee17e-9974-4e4c-8065-62ef472cd371 is finished
2021-06-06 23:17:33,195 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(47609120-fd98-4ec4-acf2-d347b5a784c0, queue=dev-awx-0)
[2021-06-06 23:17:33,195: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(47609120-fd98-4ec4-acf2-d347b5a784c0, queue=dev-awx-0)
2021-06-06 23:17:33,228 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f4ccbb4b-0251-4a7d-82e2-4292b1cc50ae, queue=awx_private_queue)
[2021-06-06 23:17:33,228: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f4ccbb4b-0251-4a7d-82e2-4292b1cc50ae, queue=awx_private_queue)
2021-06-06 23:17:33,230 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 82, in _execute
return self.cursor.execute(sql)
psycopg2.errors.AdminShutdown: terminating connection due to unexpected postmaster exit
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 123, in register
with advisory_lock('instance_registration_%s' % hostname):
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 66, in advisory_lock
cursor.execute(command)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 82, in _execute
return self.cursor.execute(sql)
django.db.utils.OperationalError: terminating connection due to unexpected postmaster exit
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2021-06-06 23:17:33,261 DEBUG awx.main.dispatch delivered f4ccbb4b-0251-4a7d-82e2-4292b1cc50ae to worker[224] qsize 0
2021-06-06 23:17:33,269 DEBUG awx.main.dispatch task f4ccbb4b-0251-4a7d-82e2-4292b1cc50ae starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:17:33,296 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:17:33,297 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:17:33,304 DEBUG awx.main.dispatch task f4ccbb4b-0251-4a7d-82e2-4292b1cc50ae is finished
2021-06-06 23:17:48,781 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f1085964-27f5-4531-955c-bf62087d1cab, queue=awx_private_queue)
[2021-06-06 23:17:48,781: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f1085964-27f5-4531-955c-bf62087d1cab, queue=awx_private_queue)
2021-06-06 23:17:48,807 DEBUG awx.main.dispatch delivered f1085964-27f5-4531-955c-bf62087d1cab to worker[215] qsize 0
2021-06-06 23:17:48,816 DEBUG awx.main.dispatch task f1085964-27f5-4531-955c-bf62087d1cab starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:17:48,844 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:17:48,847 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:17:48,853 DEBUG awx.main.dispatch task f1085964-27f5-4531-955c-bf62087d1cab is finished
2021-06-06 23:17:53,159 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(50ae064c-86f0-468d-8709-ab584b104a9b, queue=dev-awx-0)
[2021-06-06 23:17:53,159: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(50ae064c-86f0-468d-8709-ab584b104a9b, queue=dev-awx-0)
2021-06-06 23:17:53,183 DEBUG awx.main.dispatch delivered 50ae064c-86f0-468d-8709-ab584b104a9b to worker[224] qsize 0
2021-06-06 23:17:53,190 DEBUG awx.main.dispatch task 50ae064c-86f0-468d-8709-ab584b104a9b starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:17:53,207 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139824236545600_1" does not exist
2021-06-06 23:17:53,211 DEBUG awx.main.dispatch task 50ae064c-86f0-468d-8709-ab584b104a9b is finished
2021-06-06 23:17:53,228 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0957ba38-973e-418c-9ee1-c76fcccae0a5, queue=awx_private_queue)
[2021-06-06 23:17:53,228: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0957ba38-973e-418c-9ee1-c76fcccae0a5, queue=awx_private_queue)
2021-06-06 23:17:53,251 DEBUG awx.main.dispatch delivered 0957ba38-973e-418c-9ee1-c76fcccae0a5 to worker[215] qsize 0
2021-06-06 23:17:53,262 DEBUG awx.main.dispatch task 0957ba38-973e-418c-9ee1-c76fcccae0a5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:17:53,274 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:17:53,275 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:17:53,278 DEBUG awx.main.dispatch task 0957ba38-973e-418c-9ee1-c76fcccae0a5 is finished
RESULT 2
OKREADY
2021-06-06 23:18:11,409 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(d55c85f6-b7dc-435d-a98a-93c4f1edb986, queue=dev-awx-0)
[2021-06-06 23:18:11,409: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(d55c85f6-b7dc-435d-a98a-93c4f1edb986, queue=dev-awx-0)
2021-06-06 23:18:11,451 DEBUG awx.main.dispatch delivered d55c85f6-b7dc-435d-a98a-93c4f1edb986 to worker[219] qsize 0
2021-06-06 23:18:11,470 DEBUG awx.main.dispatch task d55c85f6-b7dc-435d-a98a-93c4f1edb986 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-06-06 23:18:11,497 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-06-06 23:18:11,510 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 470, in awx_isolated_heartbeat
for isolated_instance in isolated_instance_qs:
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-06-06 23:18:11,516 DEBUG awx.main.dispatch task d55c85f6-b7dc-435d-a98a-93c4f1edb986 is finished
2021-06-06 23:18:13,228 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(89cd3e7b-f827-4597-8bf2-4b82c952f37d, queue=awx_private_queue)
[2021-06-06 23:18:13,228: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(89cd3e7b-f827-4597-8bf2-4b82c952f37d, queue=awx_private_queue)
2021-06-06 23:18:13,253 DEBUG awx.main.dispatch delivered 89cd3e7b-f827-4597-8bf2-4b82c952f37d to worker[224] qsize 0
2021-06-06 23:18:13,266 DEBUG awx.main.dispatch task 89cd3e7b-f827-4597-8bf2-4b82c952f37d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:18:13,279 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:18:13,281 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:18:13,285 DEBUG awx.main.dispatch task 89cd3e7b-f827-4597-8bf2-4b82c952f37d is finished
2021-06-06 23:18:18,781 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6ee81208-23fb-4d83-a4ff-44f9c44fd469, queue=awx_private_queue)
[2021-06-06 23:18:18,781: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6ee81208-23fb-4d83-a4ff-44f9c44fd469, queue=awx_private_queue)
2021-06-06 23:18:18,814 DEBUG awx.main.dispatch delivered 6ee81208-23fb-4d83-a4ff-44f9c44fd469 to worker[219] qsize 0
2021-06-06 23:18:18,821 DEBUG awx.main.dispatch task 6ee81208-23fb-4d83-a4ff-44f9c44fd469 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:18:18,836 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:18:18,839 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:18:18,846 DEBUG awx.main.dispatch task 6ee81208-23fb-4d83-a4ff-44f9c44fd469 is finished
2021-06-06 23:18:33,202 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(72ae84c8-fed4-4cb5-a490-f14e2153bee6, queue=dev-awx-0)
[2021-06-06 23:18:33,202: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(72ae84c8-fed4-4cb5-a490-f14e2153bee6, queue=dev-awx-0)
2021-06-06 23:18:33,231 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c581cbdf-aa9f-48e6-bac2-d32c8c45e143, queue=awx_private_queue)
[2021-06-06 23:18:33,231: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c581cbdf-aa9f-48e6-bac2-d32c8c45e143, queue=awx_private_queue)
2021-06-06 23:18:33,240 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 23:18:33,260 DEBUG awx.main.dispatch delivered c581cbdf-aa9f-48e6-bac2-d32c8c45e143 to worker[215] qsize 0
2021-06-06 23:18:33,269 DEBUG awx.main.dispatch task c581cbdf-aa9f-48e6-bac2-d32c8c45e143 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:18:33,286 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:18:33,288 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:18:33,292 DEBUG awx.main.dispatch task c581cbdf-aa9f-48e6-bac2-d32c8c45e143 is finished
2021-06-06 23:18:48,786 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a54b397a-00ba-458f-ba42-46c28e3d65e5, queue=awx_private_queue)
[2021-06-06 23:18:48,786: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a54b397a-00ba-458f-ba42-46c28e3d65e5, queue=awx_private_queue)
2021-06-06 23:18:48,811 DEBUG awx.main.dispatch delivered a54b397a-00ba-458f-ba42-46c28e3d65e5 to worker[224] qsize 0
2021-06-06 23:18:48,819 DEBUG awx.main.dispatch task a54b397a-00ba-458f-ba42-46c28e3d65e5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:18:48,837 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:18:48,840 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:18:48,844 DEBUG awx.main.dispatch task a54b397a-00ba-458f-ba42-46c28e3d65e5 is finished
2021-06-06 23:18:53,158 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c8e9afa8-cf03-4212-8d09-798ff2dff070, queue=dev-awx-0)
[2021-06-06 23:18:53,158: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c8e9afa8-cf03-4212-8d09-798ff2dff070, queue=dev-awx-0)
2021-06-06 23:18:53,180 DEBUG awx.main.dispatch delivered c8e9afa8-cf03-4212-8d09-798ff2dff070 to worker[213] qsize 0
2021-06-06 23:18:53,189 DEBUG awx.main.dispatch task c8e9afa8-cf03-4212-8d09-798ff2dff070 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:18:53,221 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139824236545600_2" does not exist
2021-06-06 23:18:53,226 DEBUG awx.main.dispatch task c8e9afa8-cf03-4212-8d09-798ff2dff070 is finished
2021-06-06 23:18:53,230 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(716fa54c-31f5-4b45-8d32-8967a87091e9, queue=awx_private_queue)
[2021-06-06 23:18:53,230: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(716fa54c-31f5-4b45-8d32-8967a87091e9, queue=awx_private_queue)
2021-06-06 23:18:53,254 DEBUG awx.main.dispatch delivered 716fa54c-31f5-4b45-8d32-8967a87091e9 to worker[219] qsize 0
2021-06-06 23:18:53,274 DEBUG awx.main.dispatch task 716fa54c-31f5-4b45-8d32-8967a87091e9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:18:53,291 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:18:53,293 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:18:53,297 DEBUG awx.main.dispatch task 716fa54c-31f5-4b45-8d32-8967a87091e9 is finished
RESULT 2
OKREADY
2021-06-06 23:19:13,240 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d8dfcbf2-4c02-4d36-9efd-8265f558f1cc, queue=awx_private_queue)
[2021-06-06 23:19:13,240: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d8dfcbf2-4c02-4d36-9efd-8265f558f1cc, queue=awx_private_queue)
2021-06-06 23:19:13,266 DEBUG awx.main.dispatch delivered d8dfcbf2-4c02-4d36-9efd-8265f558f1cc to worker[215] qsize 0
2021-06-06 23:19:13,274 DEBUG awx.main.dispatch task d8dfcbf2-4c02-4d36-9efd-8265f558f1cc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:19:13,291 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:19:13,293 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:19:13,299 DEBUG awx.main.dispatch task d8dfcbf2-4c02-4d36-9efd-8265f558f1cc is finished
2021-06-06 23:19:18,787 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a640d64b-0db7-448c-8423-048f14f8fa07, queue=awx_private_queue)
[2021-06-06 23:19:18,787: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a640d64b-0db7-448c-8423-048f14f8fa07, queue=awx_private_queue)
2021-06-06 23:19:18,829 DEBUG awx.main.dispatch delivered a640d64b-0db7-448c-8423-048f14f8fa07 to worker[224] qsize 0
2021-06-06 23:19:18,837 DEBUG awx.main.dispatch task a640d64b-0db7-448c-8423-048f14f8fa07 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:19:18,856 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:19:18,859 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:19:18,863 DEBUG awx.main.dispatch task a640d64b-0db7-448c-8423-048f14f8fa07 is finished
2021-06-06 23:19:33,207 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(75a5db83-441e-48b7-8c8f-164d3c8e406d, queue=dev-awx-0)
[2021-06-06 23:19:33,207: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(75a5db83-441e-48b7-8c8f-164d3c8e406d, queue=dev-awx-0)
2021-06-06 23:19:33,247 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8c74fc8c-680e-4aa5-a38b-aad20f1a459d, queue=awx_private_queue)
[2021-06-06 23:19:33,247: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8c74fc8c-680e-4aa5-a38b-aad20f1a459d, queue=awx_private_queue)
2021-06-06 23:19:33,253 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 23:19:33,275 DEBUG awx.main.dispatch delivered 8c74fc8c-680e-4aa5-a38b-aad20f1a459d to worker[215] qsize 0
2021-06-06 23:19:33,281 DEBUG awx.main.dispatch task 8c74fc8c-680e-4aa5-a38b-aad20f1a459d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:19:33,293 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:19:33,295 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:19:33,299 DEBUG awx.main.dispatch task 8c74fc8c-680e-4aa5-a38b-aad20f1a459d is finished
2021-06-06 23:19:48,805 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7760dea1-a67d-4165-8c29-0bdcf35daa2a, queue=awx_private_queue)
[2021-06-06 23:19:48,805: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7760dea1-a67d-4165-8c29-0bdcf35daa2a, queue=awx_private_queue)
2021-06-06 23:19:48,848 DEBUG awx.main.dispatch delivered 7760dea1-a67d-4165-8c29-0bdcf35daa2a to worker[215] qsize 0
2021-06-06 23:19:48,863 DEBUG awx.main.dispatch task 7760dea1-a67d-4165-8c29-0bdcf35daa2a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:19:48,878 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:19:48,881 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:19:48,885 DEBUG awx.main.dispatch task 7760dea1-a67d-4165-8c29-0bdcf35daa2a is finished
2021-06-06 23:19:53,153 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(0f8d5a89-537b-4e9a-9a2e-2a69fe3c72b8, queue=dev-awx-0)
[2021-06-06 23:19:53,153: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(0f8d5a89-537b-4e9a-9a2e-2a69fe3c72b8, queue=dev-awx-0)
2021-06-06 23:19:53,177 DEBUG awx.main.dispatch delivered 0f8d5a89-537b-4e9a-9a2e-2a69fe3c72b8 to worker[213] qsize 0
2021-06-06 23:19:53,186 DEBUG awx.main.dispatch task 0f8d5a89-537b-4e9a-9a2e-2a69fe3c72b8 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:19:53,203 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139824236545600_3" does not exist
2021-06-06 23:19:53,211 DEBUG awx.main.dispatch task 0f8d5a89-537b-4e9a-9a2e-2a69fe3c72b8 is finished
2021-06-06 23:19:53,248 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(51fb4c9d-e7c4-40a0-9262-ecb2ed089d68, queue=awx_private_queue)
[2021-06-06 23:19:53,248: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(51fb4c9d-e7c4-40a0-9262-ecb2ed089d68, queue=awx_private_queue)
2021-06-06 23:19:53,289 DEBUG awx.main.dispatch delivered 51fb4c9d-e7c4-40a0-9262-ecb2ed089d68 to worker[219] qsize 0
2021-06-06 23:19:53,296 DEBUG awx.main.dispatch task 51fb4c9d-e7c4-40a0-9262-ecb2ed089d68 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:19:53,316 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:19:53,318 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:19:53,325 DEBUG awx.main.dispatch task 51fb4c9d-e7c4-40a0-9262-ecb2ed089d68 is finished
RESULT 2
OKREADY
2021-06-06 23:20:13,248 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(59ba0e21-88f9-455f-a555-5145cea9a430, queue=awx_private_queue)
[2021-06-06 23:20:13,248: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(59ba0e21-88f9-455f-a555-5145cea9a430, queue=awx_private_queue)
2021-06-06 23:20:13,292 DEBUG awx.main.dispatch delivered 59ba0e21-88f9-455f-a555-5145cea9a430 to worker[224] qsize 0
2021-06-06 23:20:13,301 DEBUG awx.main.dispatch task 59ba0e21-88f9-455f-a555-5145cea9a430 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:20:13,316 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:20:13,317 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:20:13,322 DEBUG awx.main.dispatch task 59ba0e21-88f9-455f-a555-5145cea9a430 is finished
2021-06-06 23:20:18,794 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e1ed66b6-78dd-489a-a189-51d03c70c9de, queue=awx_private_queue)
[2021-06-06 23:20:18,794: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e1ed66b6-78dd-489a-a189-51d03c70c9de, queue=awx_private_queue)
2021-06-06 23:20:18,819 DEBUG awx.main.dispatch delivered e1ed66b6-78dd-489a-a189-51d03c70c9de to worker[224] qsize 0
2021-06-06 23:20:18,825 DEBUG awx.main.dispatch task e1ed66b6-78dd-489a-a189-51d03c70c9de starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:20:18,838 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:20:18,840 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:20:18,845 DEBUG awx.main.dispatch task e1ed66b6-78dd-489a-a189-51d03c70c9de is finished
2021-06-06 23:20:33,207 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(fe35dc04-0372-4950-9c35-208efba3a979, queue=dev-awx-0)
[2021-06-06 23:20:33,207: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(fe35dc04-0372-4950-9c35-208efba3a979, queue=dev-awx-0)
2021-06-06 23:20:33,245 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 23:20:33,247 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d6c25166-1142-4c8e-872f-3d7c644b632a, queue=awx_private_queue)
[2021-06-06 23:20:33,247: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d6c25166-1142-4c8e-872f-3d7c644b632a, queue=awx_private_queue)
2021-06-06 23:20:33,271 DEBUG awx.main.dispatch delivered d6c25166-1142-4c8e-872f-3d7c644b632a to worker[224] qsize 0
2021-06-06 23:20:33,281 DEBUG awx.main.dispatch task d6c25166-1142-4c8e-872f-3d7c644b632a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:20:33,295 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:20:33,297 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:20:33,303 DEBUG awx.main.dispatch task d6c25166-1142-4c8e-872f-3d7c644b632a is finished
2021-06-06 23:20:48,796 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(17748fa2-59a6-40ef-af6b-57f8b95beb33, queue=awx_private_queue)
[2021-06-06 23:20:48,796: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(17748fa2-59a6-40ef-af6b-57f8b95beb33, queue=awx_private_queue)
2021-06-06 23:20:48,825 DEBUG awx.main.dispatch delivered 17748fa2-59a6-40ef-af6b-57f8b95beb33 to worker[213] qsize 0
2021-06-06 23:20:48,833 DEBUG awx.main.dispatch task 17748fa2-59a6-40ef-af6b-57f8b95beb33 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:20:48,850 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:20:48,853 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:20:48,858 DEBUG awx.main.dispatch task 17748fa2-59a6-40ef-af6b-57f8b95beb33 is finished
2021-06-06 23:20:53,153 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8163e150-f247-448c-b94a-f140dfa7aa89, queue=dev-awx-0)
[2021-06-06 23:20:53,153: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8163e150-f247-448c-b94a-f140dfa7aa89, queue=dev-awx-0)
2021-06-06 23:20:53,177 DEBUG awx.main.dispatch delivered 8163e150-f247-448c-b94a-f140dfa7aa89 to worker[224] qsize 0
2021-06-06 23:20:53,184 DEBUG awx.main.dispatch task 8163e150-f247-448c-b94a-f140dfa7aa89 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:20:53,199 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139824236545600_2" does not exist
2021-06-06 23:20:53,203 DEBUG awx.main.dispatch task 8163e150-f247-448c-b94a-f140dfa7aa89 is finished
2021-06-06 23:20:53,248 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fd857999-22d0-4729-b51c-225694bc8b9b, queue=awx_private_queue)
[2021-06-06 23:20:53,248: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fd857999-22d0-4729-b51c-225694bc8b9b, queue=awx_private_queue)
2021-06-06 23:20:53,277 DEBUG awx.main.dispatch delivered fd857999-22d0-4729-b51c-225694bc8b9b to worker[215] qsize 0
2021-06-06 23:20:53,284 DEBUG awx.main.dispatch task fd857999-22d0-4729-b51c-225694bc8b9b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:20:53,304 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:20:53,306 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:20:53,311 DEBUG awx.main.dispatch task fd857999-22d0-4729-b51c-225694bc8b9b is finished
RESULT 2
OKREADY
2021-06-06 23:21:13,263 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3d556084-61b4-4584-9bec-d22d8d4a6f4d, queue=awx_private_queue)
[2021-06-06 23:21:13,263: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3d556084-61b4-4584-9bec-d22d8d4a6f4d, queue=awx_private_queue)
2021-06-06 23:21:13,294 DEBUG awx.main.dispatch delivered 3d556084-61b4-4584-9bec-d22d8d4a6f4d to worker[215] qsize 0
2021-06-06 23:21:13,302 DEBUG awx.main.dispatch task 3d556084-61b4-4584-9bec-d22d8d4a6f4d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:21:13,316 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:21:13,318 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:21:13,322 DEBUG awx.main.dispatch task 3d556084-61b4-4584-9bec-d22d8d4a6f4d is finished
2021-06-06 23:21:18,796 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f2f439c3-505e-466b-8cec-a529cf6fa38c, queue=awx_private_queue)
[2021-06-06 23:21:18,796: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f2f439c3-505e-466b-8cec-a529cf6fa38c, queue=awx_private_queue)
2021-06-06 23:21:18,818 DEBUG awx.main.dispatch delivered f2f439c3-505e-466b-8cec-a529cf6fa38c to worker[224] qsize 0
2021-06-06 23:21:18,824 DEBUG awx.main.dispatch task f2f439c3-505e-466b-8cec-a529cf6fa38c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:21:18,836 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:21:18,838 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:21:18,843 DEBUG awx.main.dispatch task f2f439c3-505e-466b-8cec-a529cf6fa38c is finished
2021-06-06 23:21:33,213 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f36ad5e0-94c7-400b-98b9-f5533bf61acd, queue=dev-awx-0)
[2021-06-06 23:21:33,213: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f36ad5e0-94c7-400b-98b9-f5533bf61acd, queue=dev-awx-0)
2021-06-06 23:21:33,264 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d12f4aca-5ee0-4d42-9c6a-3ac1c7228d1d, queue=awx_private_queue)
[2021-06-06 23:21:33,264: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d12f4aca-5ee0-4d42-9c6a-3ac1c7228d1d, queue=awx_private_queue)
2021-06-06 23:21:33,264 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 23:21:33,288 DEBUG awx.main.dispatch delivered d12f4aca-5ee0-4d42-9c6a-3ac1c7228d1d to worker[213] qsize 0
2021-06-06 23:21:33,296 DEBUG awx.main.dispatch task d12f4aca-5ee0-4d42-9c6a-3ac1c7228d1d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:21:33,313 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:21:33,315 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:21:33,319 DEBUG awx.main.dispatch task d12f4aca-5ee0-4d42-9c6a-3ac1c7228d1d is finished
2021-06-06 23:21:48,797 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ba616f2d-828e-4380-a123-7fcdd1c489b4, queue=awx_private_queue)
[2021-06-06 23:21:48,797: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ba616f2d-828e-4380-a123-7fcdd1c489b4, queue=awx_private_queue)
2021-06-06 23:21:48,823 DEBUG awx.main.dispatch delivered ba616f2d-828e-4380-a123-7fcdd1c489b4 to worker[213] qsize 0
2021-06-06 23:21:48,831 DEBUG awx.main.dispatch task ba616f2d-828e-4380-a123-7fcdd1c489b4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:21:48,846 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:21:48,850 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:21:48,857 DEBUG awx.main.dispatch task ba616f2d-828e-4380-a123-7fcdd1c489b4 is finished
2021-06-06 23:21:53,168 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(48398fbe-fd24-4781-93e4-57123440ef7a, queue=dev-awx-0)
[2021-06-06 23:21:53,168: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(48398fbe-fd24-4781-93e4-57123440ef7a, queue=dev-awx-0)
2021-06-06 23:21:53,191 DEBUG awx.main.dispatch delivered 48398fbe-fd24-4781-93e4-57123440ef7a to worker[213] qsize 0
2021-06-06 23:21:53,197 DEBUG awx.main.dispatch task 48398fbe-fd24-4781-93e4-57123440ef7a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:21:53,210 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139824236545600_4" does not exist
2021-06-06 23:21:53,216 DEBUG awx.main.dispatch task 48398fbe-fd24-4781-93e4-57123440ef7a is finished
2021-06-06 23:21:53,258 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(040a3229-267b-43e8-a39c-46e054d1e0ce, queue=awx_private_queue)
[2021-06-06 23:21:53,258: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(040a3229-267b-43e8-a39c-46e054d1e0ce, queue=awx_private_queue)
2021-06-06 23:21:53,279 DEBUG awx.main.dispatch delivered 040a3229-267b-43e8-a39c-46e054d1e0ce to worker[215] qsize 0
2021-06-06 23:21:53,286 DEBUG awx.main.dispatch task 040a3229-267b-43e8-a39c-46e054d1e0ce starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:21:53,303 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:21:53,304 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:21:53,308 DEBUG awx.main.dispatch task 040a3229-267b-43e8-a39c-46e054d1e0ce is finished
RESULT 2
OKREADY
2021-06-06 23:22:13,262 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2374eb9b-ae37-48ca-8723-5db7361b3e49, queue=awx_private_queue)
[2021-06-06 23:22:13,262: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2374eb9b-ae37-48ca-8723-5db7361b3e49, queue=awx_private_queue)
2021-06-06 23:22:13,286 DEBUG awx.main.dispatch delivered 2374eb9b-ae37-48ca-8723-5db7361b3e49 to worker[213] qsize 0
2021-06-06 23:22:13,299 DEBUG awx.main.dispatch task 2374eb9b-ae37-48ca-8723-5db7361b3e49 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:22:13,317 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:22:13,321 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:22:13,326 DEBUG awx.main.dispatch task 2374eb9b-ae37-48ca-8723-5db7361b3e49 is finished
2021-06-06 23:22:18,797 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(daa3704d-b298-4e45-a10d-04885187e71f, queue=awx_private_queue)
[2021-06-06 23:22:18,797: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(daa3704d-b298-4e45-a10d-04885187e71f, queue=awx_private_queue)
2021-06-06 23:22:18,827 DEBUG awx.main.dispatch delivered daa3704d-b298-4e45-a10d-04885187e71f to worker[224] qsize 0
2021-06-06 23:22:18,835 DEBUG awx.main.dispatch task daa3704d-b298-4e45-a10d-04885187e71f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:22:18,853 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:22:18,856 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:22:18,863 DEBUG awx.main.dispatch task daa3704d-b298-4e45-a10d-04885187e71f is finished
2021-06-06 23:22:33,208 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(9e88de0e-3fc9-4330-b9f2-e65d6dcd6239, queue=dev-awx-0)
[2021-06-06 23:22:33,208: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(9e88de0e-3fc9-4330-b9f2-e65d6dcd6239, queue=dev-awx-0)
2021-06-06 23:22:33,237 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 23:22:33,264 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9dddee29-2ada-4630-9964-5d67e830b65b, queue=awx_private_queue)
[2021-06-06 23:22:33,264: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9dddee29-2ada-4630-9964-5d67e830b65b, queue=awx_private_queue)
2021-06-06 23:22:33,306 DEBUG awx.main.dispatch delivered 9dddee29-2ada-4630-9964-5d67e830b65b to worker[219] qsize 0
2021-06-06 23:22:33,321 DEBUG awx.main.dispatch task 9dddee29-2ada-4630-9964-5d67e830b65b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:22:33,340 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:22:33,342 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:22:33,347 DEBUG awx.main.dispatch task 9dddee29-2ada-4630-9964-5d67e830b65b is finished
2021-06-06 23:22:48,801 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e3e26eb9-0f8a-4a01-8dff-3616e6641a58, queue=awx_private_queue)
[2021-06-06 23:22:48,801: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e3e26eb9-0f8a-4a01-8dff-3616e6641a58, queue=awx_private_queue)
2021-06-06 23:22:48,827 DEBUG awx.main.dispatch delivered e3e26eb9-0f8a-4a01-8dff-3616e6641a58 to worker[215] qsize 0
2021-06-06 23:22:48,835 DEBUG awx.main.dispatch task e3e26eb9-0f8a-4a01-8dff-3616e6641a58 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:22:48,859 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:22:48,862 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:22:48,867 DEBUG awx.main.dispatch task e3e26eb9-0f8a-4a01-8dff-3616e6641a58 is finished
2021-06-06 23:22:53,159 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(75325b36-3d9a-45aa-9934-7900a1c2611b, queue=dev-awx-0)
[2021-06-06 23:22:53,159: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(75325b36-3d9a-45aa-9934-7900a1c2611b, queue=dev-awx-0)
2021-06-06 23:22:53,183 DEBUG awx.main.dispatch delivered 75325b36-3d9a-45aa-9934-7900a1c2611b to worker[224] qsize 0
2021-06-06 23:22:53,190 DEBUG awx.main.dispatch task 75325b36-3d9a-45aa-9934-7900a1c2611b starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:22:53,204 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139824236545600_3" does not exist
2021-06-06 23:22:53,208 DEBUG awx.main.dispatch task 75325b36-3d9a-45aa-9934-7900a1c2611b is finished
2021-06-06 23:22:53,261 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(19865d91-3a2f-411a-aa23-1796f97e01b3, queue=awx_private_queue)
[2021-06-06 23:22:53,261: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(19865d91-3a2f-411a-aa23-1796f97e01b3, queue=awx_private_queue)
2021-06-06 23:22:53,289 DEBUG awx.main.dispatch delivered 19865d91-3a2f-411a-aa23-1796f97e01b3 to worker[219] qsize 0
2021-06-06 23:22:53,297 DEBUG awx.main.dispatch task 19865d91-3a2f-411a-aa23-1796f97e01b3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:22:53,311 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:22:53,312 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:22:53,317 DEBUG awx.main.dispatch task 19865d91-3a2f-411a-aa23-1796f97e01b3 is finished
RESULT 2
OKREADY
2021-06-06 23:23:13,276 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f312f460-d133-473a-9087-0f827e618dc3, queue=awx_private_queue)
[2021-06-06 23:23:13,276: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f312f460-d133-473a-9087-0f827e618dc3, queue=awx_private_queue)
2021-06-06 23:23:13,313 DEBUG awx.main.dispatch delivered f312f460-d133-473a-9087-0f827e618dc3 to worker[213] qsize 0
2021-06-06 23:23:13,324 DEBUG awx.main.dispatch task f312f460-d133-473a-9087-0f827e618dc3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:23:13,343 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:23:13,344 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:23:13,350 DEBUG awx.main.dispatch task f312f460-d133-473a-9087-0f827e618dc3 is finished
2021-06-06 23:23:18,801 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(091d5454-8f1f-471f-8105-1c48fc7b204d, queue=awx_private_queue)
[2021-06-06 23:23:18,801: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(091d5454-8f1f-471f-8105-1c48fc7b204d, queue=awx_private_queue)
2021-06-06 23:23:18,830 DEBUG awx.main.dispatch delivered 091d5454-8f1f-471f-8105-1c48fc7b204d to worker[213] qsize 0
2021-06-06 23:23:18,836 DEBUG awx.main.dispatch task 091d5454-8f1f-471f-8105-1c48fc7b204d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:23:18,850 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:23:18,853 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:23:18,859 DEBUG awx.main.dispatch task 091d5454-8f1f-471f-8105-1c48fc7b204d is finished
2021-06-06 23:23:33,213 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(381918c8-4411-4b9e-9e70-23ef7543dedb, queue=dev-awx-0)
[2021-06-06 23:23:33,213: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(381918c8-4411-4b9e-9e70-23ef7543dedb, queue=dev-awx-0)
2021-06-06 23:23:33,262 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6a00c6dd-9eb1-402f-9b62-726579d08e2e, queue=awx_private_queue)
[2021-06-06 23:23:33,262: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6a00c6dd-9eb1-402f-9b62-726579d08e2e, queue=awx_private_queue)
2021-06-06 23:23:33,262 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 23:23:33,298 DEBUG awx.main.dispatch delivered 6a00c6dd-9eb1-402f-9b62-726579d08e2e to worker[224] qsize 0
2021-06-06 23:23:33,306 DEBUG awx.main.dispatch task 6a00c6dd-9eb1-402f-9b62-726579d08e2e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:23:33,323 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:23:33,325 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:23:33,330 DEBUG awx.main.dispatch task 6a00c6dd-9eb1-402f-9b62-726579d08e2e is finished
2021-06-06 23:23:48,807 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2ae877b9-7368-4ce4-b8ec-d146b1c64a3b, queue=awx_private_queue)
[2021-06-06 23:23:48,807: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2ae877b9-7368-4ce4-b8ec-d146b1c64a3b, queue=awx_private_queue)
2021-06-06 23:23:48,839 DEBUG awx.main.dispatch delivered 2ae877b9-7368-4ce4-b8ec-d146b1c64a3b to worker[224] qsize 0
2021-06-06 23:23:48,861 DEBUG awx.main.dispatch task 2ae877b9-7368-4ce4-b8ec-d146b1c64a3b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:23:48,879 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:23:48,882 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:23:48,886 DEBUG awx.main.dispatch task 2ae877b9-7368-4ce4-b8ec-d146b1c64a3b is finished
2021-06-06 23:23:53,153 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(e89df6c6-e3a3-4260-a066-2fde451bbbfd, queue=dev-awx-0)
[2021-06-06 23:23:53,153: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(e89df6c6-e3a3-4260-a066-2fde451bbbfd, queue=dev-awx-0)
2021-06-06 23:23:53,182 DEBUG awx.main.dispatch delivered e89df6c6-e3a3-4260-a066-2fde451bbbfd to worker[213] qsize 0
2021-06-06 23:23:53,189 DEBUG awx.main.dispatch task e89df6c6-e3a3-4260-a066-2fde451bbbfd starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:23:53,212 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139824236545600_5" does not exist
2021-06-06 23:23:53,217 DEBUG awx.main.dispatch task e89df6c6-e3a3-4260-a066-2fde451bbbfd is finished
2021-06-06 23:23:53,262 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(674f702c-25d4-4e5a-83a3-8dc820a18c4f, queue=awx_private_queue)
[2021-06-06 23:23:53,262: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(674f702c-25d4-4e5a-83a3-8dc820a18c4f, queue=awx_private_queue)
2021-06-06 23:23:53,295 DEBUG awx.main.dispatch delivered 674f702c-25d4-4e5a-83a3-8dc820a18c4f to worker[213] qsize 0
2021-06-06 23:23:53,302 DEBUG awx.main.dispatch task 674f702c-25d4-4e5a-83a3-8dc820a18c4f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:23:53,313 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:23:53,314 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:23:53,319 DEBUG awx.main.dispatch task 674f702c-25d4-4e5a-83a3-8dc820a18c4f is finished
RESULT 2
OKREADY
2021-06-06 23:24:13,267 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0600563a-234f-4caf-970c-bf162f0ff044, queue=awx_private_queue)
[2021-06-06 23:24:13,267: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0600563a-234f-4caf-970c-bf162f0ff044, queue=awx_private_queue)
2021-06-06 23:24:13,295 DEBUG awx.main.dispatch delivered 0600563a-234f-4caf-970c-bf162f0ff044 to worker[224] qsize 0
2021-06-06 23:24:13,305 DEBUG awx.main.dispatch task 0600563a-234f-4caf-970c-bf162f0ff044 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:24:13,318 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:24:13,320 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:24:13,324 DEBUG awx.main.dispatch task 0600563a-234f-4caf-970c-bf162f0ff044 is finished
2021-06-06 23:24:18,806 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(cc1ec205-58ef-4a2a-93f9-a405e6971c69, queue=awx_private_queue)
[2021-06-06 23:24:18,806: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(cc1ec205-58ef-4a2a-93f9-a405e6971c69, queue=awx_private_queue)
2021-06-06 23:24:18,829 DEBUG awx.main.dispatch delivered cc1ec205-58ef-4a2a-93f9-a405e6971c69 to worker[224] qsize 0
2021-06-06 23:24:18,835 DEBUG awx.main.dispatch task cc1ec205-58ef-4a2a-93f9-a405e6971c69 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:24:18,846 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:24:18,848 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:24:18,853 DEBUG awx.main.dispatch task cc1ec205-58ef-4a2a-93f9-a405e6971c69 is finished
2021-06-06 23:24:33,217 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a16e6a0d-9b28-46e1-99fc-ebaee39f77d9, queue=dev-awx-0)
[2021-06-06 23:24:33,217: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a16e6a0d-9b28-46e1-99fc-ebaee39f77d9, queue=dev-awx-0)
2021-06-06 23:24:33,251 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 23:24:33,265 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ca7d8909-3903-4538-bf63-6f92cf509ccb, queue=awx_private_queue)
[2021-06-06 23:24:33,265: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ca7d8909-3903-4538-bf63-6f92cf509ccb, queue=awx_private_queue)
2021-06-06 23:24:33,299 DEBUG awx.main.dispatch delivered ca7d8909-3903-4538-bf63-6f92cf509ccb to worker[213] qsize 0
2021-06-06 23:24:33,307 DEBUG awx.main.dispatch task ca7d8909-3903-4538-bf63-6f92cf509ccb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:24:33,322 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:24:33,324 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:24:33,328 DEBUG awx.main.dispatch task ca7d8909-3903-4538-bf63-6f92cf509ccb is finished
2021-06-06 23:24:48,811 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(acd13060-4fc5-4be4-bf68-5aa3f1b2984f, queue=awx_private_queue)
[2021-06-06 23:24:48,811: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(acd13060-4fc5-4be4-bf68-5aa3f1b2984f, queue=awx_private_queue)
2021-06-06 23:24:48,839 DEBUG awx.main.dispatch delivered acd13060-4fc5-4be4-bf68-5aa3f1b2984f to worker[215] qsize 0
2021-06-06 23:24:48,863 DEBUG awx.main.dispatch task acd13060-4fc5-4be4-bf68-5aa3f1b2984f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:24:48,887 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:24:48,889 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:24:48,893 DEBUG awx.main.dispatch task acd13060-4fc5-4be4-bf68-5aa3f1b2984f is finished
2021-06-06 23:24:53,160 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(721bb5ee-3e1b-440b-aa0c-e17df905dfd7, queue=dev-awx-0)
[2021-06-06 23:24:53,160: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(721bb5ee-3e1b-440b-aa0c-e17df905dfd7, queue=dev-awx-0)
2021-06-06 23:24:53,200 DEBUG awx.main.dispatch delivered 721bb5ee-3e1b-440b-aa0c-e17df905dfd7 to worker[215] qsize 0
2021-06-06 23:24:53,208 DEBUG awx.main.dispatch task 721bb5ee-3e1b-440b-aa0c-e17df905dfd7 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:24:53,223 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139824236545600_2" does not exist
2021-06-06 23:24:53,228 DEBUG awx.main.dispatch task 721bb5ee-3e1b-440b-aa0c-e17df905dfd7 is finished
2021-06-06 23:24:53,266 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(243ef667-db34-4ebb-b3b1-ed6bbabb6484, queue=awx_private_queue)
[2021-06-06 23:24:53,266: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(243ef667-db34-4ebb-b3b1-ed6bbabb6484, queue=awx_private_queue)
2021-06-06 23:24:53,301 DEBUG awx.main.dispatch delivered 243ef667-db34-4ebb-b3b1-ed6bbabb6484 to worker[215] qsize 0
2021-06-06 23:24:53,307 DEBUG awx.main.dispatch task 243ef667-db34-4ebb-b3b1-ed6bbabb6484 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:24:53,318 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:24:53,319 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:24:53,323 DEBUG awx.main.dispatch task 243ef667-db34-4ebb-b3b1-ed6bbabb6484 is finished
RESULT 2
OKREADY
2021-06-06 23:25:13,278 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b1dc4fab-fceb-4bb8-b7bc-069540619260, queue=awx_private_queue)
[2021-06-06 23:25:13,278: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b1dc4fab-fceb-4bb8-b7bc-069540619260, queue=awx_private_queue)
2021-06-06 23:25:13,304 DEBUG awx.main.dispatch delivered b1dc4fab-fceb-4bb8-b7bc-069540619260 to worker[215] qsize 0
2021-06-06 23:25:13,312 DEBUG awx.main.dispatch task b1dc4fab-fceb-4bb8-b7bc-069540619260 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:25:13,328 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:25:13,330 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:25:13,334 DEBUG awx.main.dispatch task b1dc4fab-fceb-4bb8-b7bc-069540619260 is finished
2021-06-06 23:25:18,813 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4c1924fb-f8e2-4a39-bda8-b75ee0bf1dbb, queue=awx_private_queue)
[2021-06-06 23:25:18,813: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4c1924fb-f8e2-4a39-bda8-b75ee0bf1dbb, queue=awx_private_queue)
2021-06-06 23:25:18,858 DEBUG awx.main.dispatch delivered 4c1924fb-f8e2-4a39-bda8-b75ee0bf1dbb to worker[224] qsize 0
2021-06-06 23:25:18,869 DEBUG awx.main.dispatch task 4c1924fb-f8e2-4a39-bda8-b75ee0bf1dbb starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:25:18,896 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:25:18,899 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:25:18,905 DEBUG awx.main.dispatch task 4c1924fb-f8e2-4a39-bda8-b75ee0bf1dbb is finished
2021-06-06 23:25:33,222 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(31ead54b-ab01-4ca0-b674-f337e30e512d, queue=dev-awx-0)
[2021-06-06 23:25:33,222: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(31ead54b-ab01-4ca0-b674-f337e30e512d, queue=dev-awx-0)
2021-06-06 23:25:33,264 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 23:25:33,276 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7ce5806b-ff3c-4087-8fe7-053631c3582c, queue=awx_private_queue)
[2021-06-06 23:25:33,276: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7ce5806b-ff3c-4087-8fe7-053631c3582c, queue=awx_private_queue)
2021-06-06 23:25:33,303 DEBUG awx.main.dispatch delivered 7ce5806b-ff3c-4087-8fe7-053631c3582c to worker[224] qsize 0
2021-06-06 23:25:33,319 DEBUG awx.main.dispatch task 7ce5806b-ff3c-4087-8fe7-053631c3582c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:25:33,337 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:25:33,340 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:25:33,346 DEBUG awx.main.dispatch task 7ce5806b-ff3c-4087-8fe7-053631c3582c is finished
2021-06-06 23:25:48,812 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(19bbe366-ea41-411a-b3d1-67c5470019d6, queue=awx_private_queue)
[2021-06-06 23:25:48,812: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(19bbe366-ea41-411a-b3d1-67c5470019d6, queue=awx_private_queue)
2021-06-06 23:25:48,836 DEBUG awx.main.dispatch delivered 19bbe366-ea41-411a-b3d1-67c5470019d6 to worker[224] qsize 0
2021-06-06 23:25:48,860 DEBUG awx.main.dispatch task 19bbe366-ea41-411a-b3d1-67c5470019d6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:25:48,882 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:25:48,884 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 23:25:48,889 DEBUG awx.main.dispatch task 19bbe366-ea41-411a-b3d1-67c5470019d6 is finished
2021-06-06 23:25:53,153 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2b98db54-1392-4187-b5bd-1a7d85f52cb9, queue=dev-awx-0)
[2021-06-06 23:25:53,153: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2b98db54-1392-4187-b5bd-1a7d85f52cb9, queue=dev-awx-0)
2021-06-06 23:25:53,176 DEBUG awx.main.dispatch delivered 2b98db54-1392-4187-b5bd-1a7d85f52cb9 to worker[215] qsize 0
2021-06-06 23:25:53,194 DEBUG awx.main.dispatch task 2b98db54-1392-4187-b5bd-1a7d85f52cb9 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 23:25:53,222 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139824236545600_3" does not exist
2021-06-06 23:25:53,227 DEBUG awx.main.dispatch task 2b98db54-1392-4187-b5bd-1a7d85f52cb9 is finished
2021-06-06 23:25:53,277 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(57a8d466-02e4-4a65-b1d8-f523a17e6d7c, queue=awx_private_queue)
[2021-06-06 23:25:53,277: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(57a8d466-02e4-4a65-b1d8-f523a17e6d7c, queue=awx_private_queue)
2021-06-06 23:25:53,299 DEBUG awx.main.dispatch delivered 57a8d466-02e4-4a65-b1d8-f523a17e6d7c to worker[213] qsize 0
2021-06-06 23:25:53,309 DEBUG awx.main.dispatch task 57a8d466-02e4-4a65-b1d8-f523a17e6d7c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:25:53,344 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:25:53,346 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:25:53,351 DEBUG awx.main.dispatch task 57a8d466-02e4-4a65-b1d8-f523a17e6d7c is finished
RESULT 2
OKREADY
2021-06-06 23:26:13,289 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2a584d80-23f6-459b-9bb8-7b102134fd1c, queue=awx_private_queue)
[2021-06-06 23:26:13,289: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2a584d80-23f6-459b-9bb8-7b102134fd1c, queue=awx_private_queue)
2021-06-06 23:26:13,319 DEBUG awx.main.dispatch delivered 2a584d80-23f6-459b-9bb8-7b102134fd1c to worker[224] qsize 0
2021-06-06 23:26:13,329 DEBUG awx.main.dispatch task 2a584d80-23f6-459b-9bb8-7b102134fd1c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 23:26:13,343 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 23:26:13,345 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 23:26:13,350 DEBUG awx.main.dispatch task 2a584d80-23f6-459b-9bb8-7b102134fd1c is finished
2021-06-06 23:26:18,813 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dfeb5ef2-6363-41ba-826a-801c9835ae02, queue=awx_private_queue)
[2021-06-06 23:26:18,813: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dfeb5ef2-6363-41ba-826a-801c9835ae02, queue=awx_private_queue)
2021-06-06 23:26:18,838 DEBUG awx.main.dispatch delivered dfeb5ef2-6363-41ba-826a-801c9835ae02 to worker[219] qsize 0
2021-06-06 23:26:18,858 DEBUG awx.main.dispatch task dfeb5ef2-6363-41ba-826a-801c9835ae02 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 23:26:18,889 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 23:26:18,893 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/li