By type
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"elapsed": 0,
"match_groupdict": {},
"match_groups": [],
"path": null,
"port": 5432,
"search_regex": null,
"state": "started"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"db": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
}
Default organization added.
Demo Credential, Inventory, and Job Template added.
(changed: True)
2021-06-06 10:48:53,324 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(a4152f3e-89cb-4d68-a326-933b88e0b5d8, queue=awx_private_queue)
Successfully registered instance dev-awx-0
(changed: True)
2021-06-06 10:48:57,836 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(edbf2e81-e0dd-4c11-be93-e2d2e818251e, queue=awx_private_queue)
2021-06-06 10:48:58,101 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(f94cd7c6-0bd4-4b4a-8f29-65bbd6502ffa, queue=awx_private_queue)
Creating instance group tower
(changed: True)
/usr/bin/launch_awx_task.sh: line 27: unset: `#!/bin/sh': not a valid identifier
2021-06-06 10:48:58,892 INFO RPC interface 'supervisor' initialized
2021-06-06 10:48:58,892 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2021-06-06 10:48:58,893 INFO supervisord started with pid 150
2021-06-06 10:48:59,898 INFO spawned: 'awx-config-watcher' with pid 153
2021-06-06 10:48:59,904 INFO spawned: 'dispatcher' with pid 154
2021-06-06 10:48:59,908 INFO spawned: 'callback-receiver' with pid 155
2021-06-06 10:48:59,914 INFO spawned: 'channels-worker' with pid 156
READY
2021-06-06 10:49:01,031 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-06-06 10:49:01,031 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-06-06 10:49:01,032 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-06-06 10:49:01,032 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
RESULT 2
OKREADY
2021-06-06 10:49:03,160 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:175
2021-06-06 10:49:03,160 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:175
2021-06-06 10:49:03,167 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-06-06 10:49:03,167 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-06-06 10:49:03,174 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2021-06-06 10:49:03,174 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2021-06-06 10:49:03,183 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2021-06-06 10:49:03,183 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2021-06-06 10:49:03,233 ERROR celery.beat Removing corrupted schedule file '/var/lib/awx/beat.db': error(11, 'Resource temporarily unavailable')
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 485, in setup_schedule
self._store = self._open_schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 475, in _open_schedule
return self.persistence.open(self.schedule_filename, writeback=True)
File "/usr/lib64/python3.6/shelve.py", line 243, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/usr/lib64/python3.6/shelve.py", line 227, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
File "/usr/lib64/python3.6/dbm/__init__.py", line 94, in open
return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable
2021-06-06 10:49:03,278 WARNING awx.main.dispatch scaling up worker pid:180
2021-06-06 10:49:03,290 - INFO - runworker - Using single-threaded worker.
2021-06-06 10:49:03,290 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer)
2021-06-06 10:49:03,291 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive
2021-06-06 10:49:03,293 WARNING awx.main.dispatch scaling up worker pid:181
2021-06-06 10:49:03,308 WARNING awx.main.dispatch scaling up worker pid:182
2021-06-06 10:49:03,322 WARNING awx.main.dispatch scaling up worker pid:183
2021-06-06 10:49:03,330 DEBUG awx.main.tasks Syncing Schedules
2021-06-06 10:49:03,686 DEBUG awx.main.tasks Waited 0.0016057491302490234 seconds to obtain lock name: cluster_policy_lock
2021-06-06 10:49:03,718 DEBUG awx.main.tasks Total non-isolated instances:1 available for policy: 1
2021-06-06 10:49:03,718 DEBUG awx.main.tasks Policy percentage, adding Instances [1] to Group tower
2021-06-06 10:49:03,718 DEBUG awx.main.tasks Adding instances [1] to group tower
2021-06-06 10:49:03,738 DEBUG awx.main.tasks Cluster policy computation finished in 0.05148887634277344 seconds
2021-06-06 10:49:03,743 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 10:49:03,805 DEBUG awx.main.dispatch listening on ['awx_private_queue [direct]', 'dev-awx-0 [direct]', 'tower_broadcast_all_dev-awx-0 [fanout]']
2021-06-06 10:49:23,398 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b61f0f73-f838-43fa-827e-6fedd35838d5, queue=awx_private_queue)
[2021-06-06 10:49:23,398: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b61f0f73-f838-43fa-827e-6fedd35838d5, queue=awx_private_queue)
2021-06-06 10:49:23,422 DEBUG awx.main.dispatch delivered b61f0f73-f838-43fa-827e-6fedd35838d5 to worker[183] qsize 0
2021-06-06 10:49:23,426 DEBUG awx.main.dispatch task b61f0f73-f838-43fa-827e-6fedd35838d5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:49:23,428 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:49:23,450 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:49:23,489 DEBUG awx.main.dispatch task b61f0f73-f838-43fa-827e-6fedd35838d5 is finished
2021-06-06 10:49:33,373 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1e2a6b94-133f-43f6-b083-06a40a60a334, queue=awx_private_queue)
[2021-06-06 10:49:33,373: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1e2a6b94-133f-43f6-b083-06a40a60a334, queue=awx_private_queue)
2021-06-06 10:49:33,387 DEBUG awx.main.dispatch delivered 1e2a6b94-133f-43f6-b083-06a40a60a334 to worker[183] qsize 0
2021-06-06 10:49:33,390 DEBUG awx.main.dispatch task 1e2a6b94-133f-43f6-b083-06a40a60a334 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:49:33,398 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:49:33,407 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:49:33.403463+00:00
2021-06-06 10:49:33,423 DEBUG awx.main.dispatch task 1e2a6b94-133f-43f6-b083-06a40a60a334 is finished
2021-06-06 10:49:43,391 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7ee2f435-1110-415d-baf1-3af254c340b3, queue=awx_private_queue)
[2021-06-06 10:49:43,391: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7ee2f435-1110-415d-baf1-3af254c340b3, queue=awx_private_queue)
2021-06-06 10:49:43,421 DEBUG awx.main.dispatch delivered 7ee2f435-1110-415d-baf1-3af254c340b3 to worker[182] qsize 0
2021-06-06 10:49:43,431 DEBUG awx.main.dispatch task 7ee2f435-1110-415d-baf1-3af254c340b3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:49:43,436 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:49:43,481 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:49:43,540 DEBUG awx.main.dispatch task 7ee2f435-1110-415d-baf1-3af254c340b3 is finished
RESULT 2
OKREADY
2021-06-06 10:50:03,374 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(404d556b-8e26-4239-9a9b-e656f934cf8d, queue=dev-awx-0)
[2021-06-06 10:50:03,374: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(404d556b-8e26-4239-9a9b-e656f934cf8d, queue=dev-awx-0)
2021-06-06 10:50:03,632 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(022b9d40-51af-4049-a85f-78f48e185083, queue=dev-awx-0)
[2021-06-06 10:50:03,632: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(022b9d40-51af-4049-a85f-78f48e185083, queue=dev-awx-0)
2021-06-06 10:50:03,646 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f22aa115-e656-481e-9a37-353072ec3d17, queue=awx_private_queue)
[2021-06-06 10:50:03,646: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f22aa115-e656-481e-9a37-353072ec3d17, queue=awx_private_queue)
2021-06-06 10:50:03,661 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1dee826a-f035-4b40-ad29-f78cb88e3749, queue=awx_private_queue)
[2021-06-06 10:50:03,661: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1dee826a-f035-4b40-ad29-f78cb88e3749, queue=awx_private_queue)
2021-06-06 10:50:03,665 DEBUG awx.main.dispatch delivered 404d556b-8e26-4239-9a9b-e656f934cf8d to worker[181] qsize 0
2021-06-06 10:50:03,669 DEBUG awx.main.dispatch delivered 022b9d40-51af-4049-a85f-78f48e185083 to worker[180] qsize 0
2021-06-06 10:50:03,670 DEBUG awx.main.dispatch task 404d556b-8e26-4239-9a9b-e656f934cf8d starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 10:50:03,677 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 10:50:03,675 DEBUG awx.main.dispatch task 022b9d40-51af-4049-a85f-78f48e185083 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 10:50:03,690 DEBUG awx.main.dispatch delivered f22aa115-e656-481e-9a37-353072ec3d17 to worker[182] qsize 0
2021-06-06 10:50:03,692 DEBUG awx.main.dispatch delivered 1dee826a-f035-4b40-ad29-f78cb88e3749 to worker[183] qsize 0
2021-06-06 10:50:03,692 DEBUG awx.main.dispatch task f22aa115-e656-481e-9a37-353072ec3d17 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:50:03,696 DEBUG awx.main.dispatch task 1dee826a-f035-4b40-ad29-f78cb88e3749 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:50:03,701 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:50:03,702 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:50:03,701 DEBUG awx.main.dispatch task 022b9d40-51af-4049-a85f-78f48e185083 is finished
2021-06-06 10:50:03,709 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:49:33.398864+00:00
2021-06-06 10:50:03,714 DEBUG awx.main.dispatch task 404d556b-8e26-4239-9a9b-e656f934cf8d is finished
2021-06-06 10:50:03,718 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:50:03,727 DEBUG awx.main.dispatch task f22aa115-e656-481e-9a37-353072ec3d17 is finished
2021-06-06 10:50:03,762 DEBUG awx.main.dispatch task 1dee826a-f035-4b40-ad29-f78cb88e3749 is finished
2021-06-06 10:50:23,675 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(26fec431-4789-4d2b-95c4-862e99481f4c, queue=awx_private_queue)
[2021-06-06 10:50:23,675: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(26fec431-4789-4d2b-95c4-862e99481f4c, queue=awx_private_queue)
2021-06-06 10:50:23,704 DEBUG awx.main.dispatch delivered 26fec431-4789-4d2b-95c4-862e99481f4c to worker[181] qsize 0
2021-06-06 10:50:23,713 DEBUG awx.main.dispatch task 26fec431-4789-4d2b-95c4-862e99481f4c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:50:23,716 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:50:23,742 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:50:23,782 DEBUG awx.main.dispatch task 26fec431-4789-4d2b-95c4-862e99481f4c is finished
2021-06-06 10:50:33,647 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c69aa463-bcfc-408d-a7b3-9086a67c080f, queue=awx_private_queue)
[2021-06-06 10:50:33,647: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c69aa463-bcfc-408d-a7b3-9086a67c080f, queue=awx_private_queue)
2021-06-06 10:50:33,662 DEBUG awx.main.dispatch delivered c69aa463-bcfc-408d-a7b3-9086a67c080f to worker[183] qsize 0
2021-06-06 10:50:33,665 DEBUG awx.main.dispatch task c69aa463-bcfc-408d-a7b3-9086a67c080f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:50:33,678 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:50:33,683 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:50:03.702656+00:00
2021-06-06 10:50:33,700 DEBUG awx.main.dispatch task c69aa463-bcfc-408d-a7b3-9086a67c080f is finished
2021-06-06 10:50:43,670 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(28add08e-08ef-40ba-864a-e0f32ba06c54, queue=awx_private_queue)
[2021-06-06 10:50:43,670: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(28add08e-08ef-40ba-864a-e0f32ba06c54, queue=awx_private_queue)
2021-06-06 10:50:43,688 DEBUG awx.main.dispatch delivered 28add08e-08ef-40ba-864a-e0f32ba06c54 to worker[180] qsize 0
2021-06-06 10:50:43,693 DEBUG awx.main.dispatch task 28add08e-08ef-40ba-864a-e0f32ba06c54 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:50:43,696 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:50:43,723 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:50:43,768 DEBUG awx.main.dispatch task 28add08e-08ef-40ba-864a-e0f32ba06c54 is finished
RESULT 2
OKREADY
2021-06-06 10:51:03,388 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(91aa677f-5105-49fc-b8f7-dd868cb194d2, queue=dev-awx-0)
[2021-06-06 10:51:03,388: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(91aa677f-5105-49fc-b8f7-dd868cb194d2, queue=dev-awx-0)
2021-06-06 10:51:03,604 DEBUG awx.main.dispatch delivered 91aa677f-5105-49fc-b8f7-dd868cb194d2 to worker[182] qsize 0
2021-06-06 10:51:03,613 DEBUG awx.main.dispatch task 91aa677f-5105-49fc-b8f7-dd868cb194d2 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 10:51:03,617 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 10:51:03,632 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c70a7b91-2368-4aea-8084-51ce3999840f, queue=dev-awx-0)
[2021-06-06 10:51:03,632: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c70a7b91-2368-4aea-8084-51ce3999840f, queue=dev-awx-0)
2021-06-06 10:51:03,644 DEBUG awx.main.dispatch task 91aa677f-5105-49fc-b8f7-dd868cb194d2 is finished
2021-06-06 10:51:03,647 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1b6933da-d74d-4d04-8db9-986d3f98846b, queue=awx_private_queue)
[2021-06-06 10:51:03,647: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1b6933da-d74d-4d04-8db9-986d3f98846b, queue=awx_private_queue)
2021-06-06 10:51:03,647 DEBUG awx.main.dispatch delivered c70a7b91-2368-4aea-8084-51ce3999840f to worker[183] qsize 0
2021-06-06 10:51:03,652 DEBUG awx.main.dispatch task c70a7b91-2368-4aea-8084-51ce3999840f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 10:51:03,662 DEBUG awx.main.dispatch delivered 1b6933da-d74d-4d04-8db9-986d3f98846b to worker[181] qsize 0
2021-06-06 10:51:03,668 DEBUG awx.main.dispatch task c70a7b91-2368-4aea-8084-51ce3999840f is finished
2021-06-06 10:51:03,667 DEBUG awx.main.dispatch task 1b6933da-d74d-4d04-8db9-986d3f98846b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:51:03,669 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e340254d-ee59-4026-a197-dfb50f8908ba, queue=awx_private_queue)
[2021-06-06 10:51:03,669: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e340254d-ee59-4026-a197-dfb50f8908ba, queue=awx_private_queue)
2021-06-06 10:51:03,678 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:51:03,684 DEBUG awx.main.dispatch delivered e340254d-ee59-4026-a197-dfb50f8908ba to worker[180] qsize 0
2021-06-06 10:51:03,683 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:50:33.678868+00:00
2021-06-06 10:51:03,690 DEBUG awx.main.dispatch task e340254d-ee59-4026-a197-dfb50f8908ba starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:51:03,696 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:51:03,705 DEBUG awx.main.dispatch task 1b6933da-d74d-4d04-8db9-986d3f98846b is finished
2021-06-06 10:51:03,715 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:51:03,751 DEBUG awx.main.dispatch task e340254d-ee59-4026-a197-dfb50f8908ba is finished
2021-06-06 10:51:23,678 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1eb08a46-bdfa-4c04-b05f-0bb5f21dd85d, queue=awx_private_queue)
[2021-06-06 10:51:23,678: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1eb08a46-bdfa-4c04-b05f-0bb5f21dd85d, queue=awx_private_queue)
2021-06-06 10:51:23,696 DEBUG awx.main.dispatch delivered 1eb08a46-bdfa-4c04-b05f-0bb5f21dd85d to worker[182] qsize 0
2021-06-06 10:51:23,699 DEBUG awx.main.dispatch task 1eb08a46-bdfa-4c04-b05f-0bb5f21dd85d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:51:23,701 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:51:23,722 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:51:23,755 DEBUG awx.main.dispatch task 1eb08a46-bdfa-4c04-b05f-0bb5f21dd85d is finished
2021-06-06 10:51:33,650 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(24f53506-f329-474c-adc5-952a15856753, queue=awx_private_queue)
[2021-06-06 10:51:33,650: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(24f53506-f329-474c-adc5-952a15856753, queue=awx_private_queue)
2021-06-06 10:51:33,674 DEBUG awx.main.dispatch delivered 24f53506-f329-474c-adc5-952a15856753 to worker[183] qsize 0
2021-06-06 10:51:33,681 DEBUG awx.main.dispatch task 24f53506-f329-474c-adc5-952a15856753 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:51:33,694 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:51:33,698 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:51:03.679028+00:00
2021-06-06 10:51:33,712 DEBUG awx.main.dispatch task 24f53506-f329-474c-adc5-952a15856753 is finished
2021-06-06 10:51:43,680 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8614202f-fc5f-46f6-bacb-e4fc89022cba, queue=awx_private_queue)
[2021-06-06 10:51:43,680: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8614202f-fc5f-46f6-bacb-e4fc89022cba, queue=awx_private_queue)
2021-06-06 10:51:43,705 DEBUG awx.main.dispatch delivered 8614202f-fc5f-46f6-bacb-e4fc89022cba to worker[182] qsize 0
2021-06-06 10:51:43,710 DEBUG awx.main.dispatch task 8614202f-fc5f-46f6-bacb-e4fc89022cba starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:51:43,714 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:51:43,735 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:51:43,763 DEBUG awx.main.dispatch task 8614202f-fc5f-46f6-bacb-e4fc89022cba is finished
RESULT 2
OKREADY
2021-06-06 10:52:03,388 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8b34c33e-e7db-477b-a874-ecdfd4e13d8a, queue=dev-awx-0)
[2021-06-06 10:52:03,388: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8b34c33e-e7db-477b-a874-ecdfd4e13d8a, queue=dev-awx-0)
2021-06-06 10:52:03,633 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a808eeba-ada7-4a1a-976e-518cc1e26bec, queue=dev-awx-0)
[2021-06-06 10:52:03,633: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a808eeba-ada7-4a1a-976e-518cc1e26bec, queue=dev-awx-0)
2021-06-06 10:52:03,643 DEBUG awx.main.dispatch delivered 8b34c33e-e7db-477b-a874-ecdfd4e13d8a to worker[183] qsize 0
2021-06-06 10:52:03,645 DEBUG awx.main.dispatch task 8b34c33e-e7db-477b-a874-ecdfd4e13d8a starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 10:52:03,647 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 10:52:03,650 DEBUG awx.main.dispatch delivered a808eeba-ada7-4a1a-976e-518cc1e26bec to worker[181] qsize 0
2021-06-06 10:52:03,652 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a96a62da-5f40-486c-b4d0-0ee0181571eb, queue=awx_private_queue)
[2021-06-06 10:52:03,652: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a96a62da-5f40-486c-b4d0-0ee0181571eb, queue=awx_private_queue)
2021-06-06 10:52:03,654 DEBUG awx.main.dispatch task a808eeba-ada7-4a1a-976e-518cc1e26bec starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 10:52:03,667 DEBUG awx.main.dispatch delivered a96a62da-5f40-486c-b4d0-0ee0181571eb to worker[180] qsize 0
2021-06-06 10:52:03,674 DEBUG awx.main.dispatch task a96a62da-5f40-486c-b4d0-0ee0181571eb starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:52:03,678 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ee1b274c-f5c0-4a68-a34a-8ed5131707c7, queue=awx_private_queue)
[2021-06-06 10:52:03,678: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ee1b274c-f5c0-4a68-a34a-8ed5131707c7, queue=awx_private_queue)
2021-06-06 10:52:03,683 DEBUG awx.main.dispatch task 8b34c33e-e7db-477b-a874-ecdfd4e13d8a is finished
2021-06-06 10:52:03,685 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:52:03,686 DEBUG awx.main.dispatch task a808eeba-ada7-4a1a-976e-518cc1e26bec is finished
2021-06-06 10:52:03,690 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:51:33.694730+00:00
2021-06-06 10:52:03,706 DEBUG awx.main.dispatch delivered ee1b274c-f5c0-4a68-a34a-8ed5131707c7 to worker[181] qsize 0
2021-06-06 10:52:03,709 DEBUG awx.main.dispatch task a96a62da-5f40-486c-b4d0-0ee0181571eb is finished
2021-06-06 10:52:03,708 DEBUG awx.main.dispatch task ee1b274c-f5c0-4a68-a34a-8ed5131707c7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:52:03,711 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:52:03,726 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:52:03,753 DEBUG awx.main.dispatch task ee1b274c-f5c0-4a68-a34a-8ed5131707c7 is finished
2021-06-06 10:52:23,678 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1b13fb5c-adea-48b8-bba9-ea0bb5c55d5f, queue=awx_private_queue)
[2021-06-06 10:52:23,678: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1b13fb5c-adea-48b8-bba9-ea0bb5c55d5f, queue=awx_private_queue)
2021-06-06 10:52:23,693 DEBUG awx.main.dispatch delivered 1b13fb5c-adea-48b8-bba9-ea0bb5c55d5f to worker[180] qsize 0
2021-06-06 10:52:23,695 DEBUG awx.main.dispatch task 1b13fb5c-adea-48b8-bba9-ea0bb5c55d5f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:52:23,697 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:52:23,713 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:52:23,743 DEBUG awx.main.dispatch task 1b13fb5c-adea-48b8-bba9-ea0bb5c55d5f is finished
2021-06-06 10:52:33,652 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1707344c-dcd2-400c-b773-dc3cb620c088, queue=awx_private_queue)
[2021-06-06 10:52:33,652: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1707344c-dcd2-400c-b773-dc3cb620c088, queue=awx_private_queue)
2021-06-06 10:52:33,668 DEBUG awx.main.dispatch delivered 1707344c-dcd2-400c-b773-dc3cb620c088 to worker[181] qsize 0
2021-06-06 10:52:33,670 DEBUG awx.main.dispatch task 1707344c-dcd2-400c-b773-dc3cb620c088 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:52:33,680 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:52:33,683 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:52:03.686026+00:00
2021-06-06 10:52:33,694 DEBUG awx.main.dispatch task 1707344c-dcd2-400c-b773-dc3cb620c088 is finished
2021-06-06 10:52:43,678 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7ab125c8-fd75-4504-9677-a4d43cf02cdd, queue=awx_private_queue)
[2021-06-06 10:52:43,678: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7ab125c8-fd75-4504-9677-a4d43cf02cdd, queue=awx_private_queue)
2021-06-06 10:52:43,692 DEBUG awx.main.dispatch delivered 7ab125c8-fd75-4504-9677-a4d43cf02cdd to worker[180] qsize 0
2021-06-06 10:52:43,694 DEBUG awx.main.dispatch task 7ab125c8-fd75-4504-9677-a4d43cf02cdd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:52:43,695 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:52:43,710 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:52:43,737 DEBUG awx.main.dispatch task 7ab125c8-fd75-4504-9677-a4d43cf02cdd is finished
RESULT 2
OKREADY
2021-06-06 10:53:03,397 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5ec05b53-9f5f-4e43-be19-6867349bafc2, queue=dev-awx-0)
[2021-06-06 10:53:03,397: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5ec05b53-9f5f-4e43-be19-6867349bafc2, queue=dev-awx-0)
2021-06-06 10:53:03,607 DEBUG awx.main.dispatch delivered 5ec05b53-9f5f-4e43-be19-6867349bafc2 to worker[183] qsize 0
2021-06-06 10:53:03,609 DEBUG awx.main.dispatch task 5ec05b53-9f5f-4e43-be19-6867349bafc2 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 10:53:03,613 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 10:53:03,632 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(af21372d-a1f4-4a24-a091-bbb3c6d7a985, queue=dev-awx-0)
[2021-06-06 10:53:03,632: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(af21372d-a1f4-4a24-a091-bbb3c6d7a985, queue=dev-awx-0)
2021-06-06 10:53:03,645 DEBUG awx.main.dispatch task 5ec05b53-9f5f-4e43-be19-6867349bafc2 is finished
2021-06-06 10:53:03,647 DEBUG awx.main.dispatch delivered af21372d-a1f4-4a24-a091-bbb3c6d7a985 to worker[180] qsize 0
2021-06-06 10:53:03,650 DEBUG awx.main.dispatch task af21372d-a1f4-4a24-a091-bbb3c6d7a985 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 10:53:03,651 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9812f45b-cb8d-4d9a-9a20-ef9fa2f25187, queue=awx_private_queue)
[2021-06-06 10:53:03,651: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9812f45b-cb8d-4d9a-9a20-ef9fa2f25187, queue=awx_private_queue)
2021-06-06 10:53:03,663 DEBUG awx.main.dispatch task af21372d-a1f4-4a24-a091-bbb3c6d7a985 is finished
2021-06-06 10:53:03,665 DEBUG awx.main.dispatch delivered 9812f45b-cb8d-4d9a-9a20-ef9fa2f25187 to worker[180] qsize 0
2021-06-06 10:53:03,666 DEBUG awx.main.dispatch task 9812f45b-cb8d-4d9a-9a20-ef9fa2f25187 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:53:03,673 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:53:03,675 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:52:33.680581+00:00
2021-06-06 10:53:03,678 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5b5231fc-6fdd-4c61-aefa-5f6cbf2e2127, queue=awx_private_queue)
[2021-06-06 10:53:03,678: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5b5231fc-6fdd-4c61-aefa-5f6cbf2e2127, queue=awx_private_queue)
2021-06-06 10:53:03,685 DEBUG awx.main.dispatch task 9812f45b-cb8d-4d9a-9a20-ef9fa2f25187 is finished
2021-06-06 10:53:03,702 DEBUG awx.main.dispatch delivered 5b5231fc-6fdd-4c61-aefa-5f6cbf2e2127 to worker[183] qsize 0
2021-06-06 10:53:03,704 DEBUG awx.main.dispatch task 5b5231fc-6fdd-4c61-aefa-5f6cbf2e2127 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:53:03,705 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:53:03,718 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:53:03,743 DEBUG awx.main.dispatch task 5b5231fc-6fdd-4c61-aefa-5f6cbf2e2127 is finished
2021-06-06 10:53:23,686 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0e93229a-b4f2-433c-b3b8-b0504fc35a47, queue=awx_private_queue)
[2021-06-06 10:53:23,686: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0e93229a-b4f2-433c-b3b8-b0504fc35a47, queue=awx_private_queue)
2021-06-06 10:53:23,711 DEBUG awx.main.dispatch delivered 0e93229a-b4f2-433c-b3b8-b0504fc35a47 to worker[181] qsize 0
2021-06-06 10:53:23,718 DEBUG awx.main.dispatch task 0e93229a-b4f2-433c-b3b8-b0504fc35a47 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:53:23,723 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:53:23,744 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:53:23,775 DEBUG awx.main.dispatch task 0e93229a-b4f2-433c-b3b8-b0504fc35a47 is finished
2021-06-06 10:53:33,653 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(177dd596-75b7-4212-b3de-73fad4bdf5d4, queue=awx_private_queue)
[2021-06-06 10:53:33,653: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(177dd596-75b7-4212-b3de-73fad4bdf5d4, queue=awx_private_queue)
2021-06-06 10:53:33,673 DEBUG awx.main.dispatch delivered 177dd596-75b7-4212-b3de-73fad4bdf5d4 to worker[182] qsize 0
2021-06-06 10:53:33,678 DEBUG awx.main.dispatch task 177dd596-75b7-4212-b3de-73fad4bdf5d4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:53:33,691 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:53:33,695 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:53:03.673321+00:00
2021-06-06 10:53:33,708 DEBUG awx.main.dispatch task 177dd596-75b7-4212-b3de-73fad4bdf5d4 is finished
2021-06-06 10:53:43,683 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fef135ae-9b12-4239-a9f5-55f646b2f694, queue=awx_private_queue)
[2021-06-06 10:53:43,683: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fef135ae-9b12-4239-a9f5-55f646b2f694, queue=awx_private_queue)
2021-06-06 10:53:43,700 DEBUG awx.main.dispatch delivered fef135ae-9b12-4239-a9f5-55f646b2f694 to worker[181] qsize 0
2021-06-06 10:53:43,703 DEBUG awx.main.dispatch task fef135ae-9b12-4239-a9f5-55f646b2f694 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:53:43,704 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:53:43,719 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:53:43,745 DEBUG awx.main.dispatch task fef135ae-9b12-4239-a9f5-55f646b2f694 is finished
RESULT 2
OKREADY
2021-06-06 10:54:03,397 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b6727198-6471-49ca-9e71-923334f121f5, queue=dev-awx-0)
[2021-06-06 10:54:03,397: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b6727198-6471-49ca-9e71-923334f121f5, queue=dev-awx-0)
2021-06-06 10:54:03,631 DEBUG awx.main.dispatch delivered b6727198-6471-49ca-9e71-923334f121f5 to worker[181] qsize 0
2021-06-06 10:54:03,632 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(eafc2e1e-7057-4031-a54a-b57095fda865, queue=dev-awx-0)
[2021-06-06 10:54:03,632: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(eafc2e1e-7057-4031-a54a-b57095fda865, queue=dev-awx-0)
2021-06-06 10:54:03,633 DEBUG awx.main.dispatch task b6727198-6471-49ca-9e71-923334f121f5 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 10:54:03,635 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 10:54:03,650 DEBUG awx.main.dispatch delivered eafc2e1e-7057-4031-a54a-b57095fda865 to worker[183] qsize 0
2021-06-06 10:54:03,652 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1421e6bf-396a-4a59-abd1-a51cfecbfc99, queue=awx_private_queue)
[2021-06-06 10:54:03,652: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1421e6bf-396a-4a59-abd1-a51cfecbfc99, queue=awx_private_queue)
2021-06-06 10:54:03,655 DEBUG awx.main.dispatch task eafc2e1e-7057-4031-a54a-b57095fda865 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 10:54:03,663 DEBUG awx.main.dispatch task b6727198-6471-49ca-9e71-923334f121f5 is finished
2021-06-06 10:54:03,670 DEBUG awx.main.dispatch delivered 1421e6bf-396a-4a59-abd1-a51cfecbfc99 to worker[180] qsize 0
2021-06-06 10:54:03,673 DEBUG awx.main.dispatch task eafc2e1e-7057-4031-a54a-b57095fda865 is finished
2021-06-06 10:54:03,673 DEBUG awx.main.dispatch task 1421e6bf-396a-4a59-abd1-a51cfecbfc99 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:54:03,683 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e4ddfb71-ae89-4d24-807e-0eaced4dbcb3, queue=awx_private_queue)
[2021-06-06 10:54:03,683: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e4ddfb71-ae89-4d24-807e-0eaced4dbcb3, queue=awx_private_queue)
2021-06-06 10:54:03,685 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:54:03,689 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:53:33.692130+00:00
2021-06-06 10:54:03,702 DEBUG awx.main.dispatch delivered e4ddfb71-ae89-4d24-807e-0eaced4dbcb3 to worker[183] qsize 0
2021-06-06 10:54:03,703 DEBUG awx.main.dispatch task 1421e6bf-396a-4a59-abd1-a51cfecbfc99 is finished
2021-06-06 10:54:03,705 DEBUG awx.main.dispatch task e4ddfb71-ae89-4d24-807e-0eaced4dbcb3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:54:03,706 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:54:03,723 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:54:03,757 DEBUG awx.main.dispatch task e4ddfb71-ae89-4d24-807e-0eaced4dbcb3 is finished
2021-06-06 10:54:23,693 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a71ade26-1c40-4b85-b6fc-344f33af57f6, queue=awx_private_queue)
[2021-06-06 10:54:23,693: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a71ade26-1c40-4b85-b6fc-344f33af57f6, queue=awx_private_queue)
2021-06-06 10:54:23,719 DEBUG awx.main.dispatch delivered a71ade26-1c40-4b85-b6fc-344f33af57f6 to worker[181] qsize 0
2021-06-06 10:54:23,723 DEBUG awx.main.dispatch task a71ade26-1c40-4b85-b6fc-344f33af57f6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:54:23,725 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:54:23,756 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:54:23,787 DEBUG awx.main.dispatch task a71ade26-1c40-4b85-b6fc-344f33af57f6 is finished
2021-06-06 10:54:33,652 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(39f4efe2-4208-4a4d-a87e-0eb959379154, queue=awx_private_queue)
[2021-06-06 10:54:33,652: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(39f4efe2-4208-4a4d-a87e-0eb959379154, queue=awx_private_queue)
2021-06-06 10:54:33,671 DEBUG awx.main.dispatch delivered 39f4efe2-4208-4a4d-a87e-0eb959379154 to worker[180] qsize 0
2021-06-06 10:54:33,673 DEBUG awx.main.dispatch task 39f4efe2-4208-4a4d-a87e-0eb959379154 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:54:33,684 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:54:33,686 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:54:03.685686+00:00
2021-06-06 10:54:33,698 DEBUG awx.main.dispatch task 39f4efe2-4208-4a4d-a87e-0eb959379154 is finished
2021-06-06 10:54:43,693 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(23b0d446-74be-4446-97b2-e559bca28821, queue=awx_private_queue)
[2021-06-06 10:54:43,693: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(23b0d446-74be-4446-97b2-e559bca28821, queue=awx_private_queue)
2021-06-06 10:54:43,712 DEBUG awx.main.dispatch delivered 23b0d446-74be-4446-97b2-e559bca28821 to worker[183] qsize 0
2021-06-06 10:54:43,714 DEBUG awx.main.dispatch task 23b0d446-74be-4446-97b2-e559bca28821 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:54:43,715 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:54:43,732 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:54:43,763 DEBUG awx.main.dispatch task 23b0d446-74be-4446-97b2-e559bca28821 is finished
RESULT 2
OKREADY
2021-06-06 10:55:03,412 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f725cd3e-79ac-4a83-9905-46c29906ecbc, queue=dev-awx-0)
[2021-06-06 10:55:03,412: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f725cd3e-79ac-4a83-9905-46c29906ecbc, queue=dev-awx-0)
2021-06-06 10:55:03,633 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(cf3fcfb1-68b2-40bf-9cb8-a06e88e8f3ac, queue=dev-awx-0)
[2021-06-06 10:55:03,633: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(cf3fcfb1-68b2-40bf-9cb8-a06e88e8f3ac, queue=dev-awx-0)
2021-06-06 10:55:03,657 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bcede725-1d9e-405f-b69d-0c109fc8022c, queue=awx_private_queue)
[2021-06-06 10:55:03,657: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bcede725-1d9e-405f-b69d-0c109fc8022c, queue=awx_private_queue)
2021-06-06 10:55:03,657 DEBUG awx.main.dispatch delivered f725cd3e-79ac-4a83-9905-46c29906ecbc to worker[181] qsize 0
2021-06-06 10:55:03,662 DEBUG awx.main.dispatch task f725cd3e-79ac-4a83-9905-46c29906ecbc starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 10:55:03,665 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 10:55:03,668 DEBUG awx.main.dispatch delivered cf3fcfb1-68b2-40bf-9cb8-a06e88e8f3ac to worker[180] qsize 0
2021-06-06 10:55:03,671 DEBUG awx.main.dispatch task cf3fcfb1-68b2-40bf-9cb8-a06e88e8f3ac starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 10:55:03,680 DEBUG awx.main.dispatch delivered bcede725-1d9e-405f-b69d-0c109fc8022c to worker[183] qsize 0
2021-06-06 10:55:03,683 DEBUG awx.main.dispatch task bcede725-1d9e-405f-b69d-0c109fc8022c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:55:03,693 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bcc30a69-d00c-4065-9613-001f7d10b2ac, queue=awx_private_queue)
[2021-06-06 10:55:03,693: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bcc30a69-d00c-4065-9613-001f7d10b2ac, queue=awx_private_queue)
2021-06-06 10:55:03,694 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:55:03,694 DEBUG awx.main.dispatch task cf3fcfb1-68b2-40bf-9cb8-a06e88e8f3ac is finished
2021-06-06 10:55:03,698 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:54:33.684149+00:00
2021-06-06 10:55:03,701 DEBUG awx.main.dispatch task f725cd3e-79ac-4a83-9905-46c29906ecbc is finished
2021-06-06 10:55:03,707 DEBUG awx.main.dispatch delivered bcc30a69-d00c-4065-9613-001f7d10b2ac to worker[180] qsize 0
2021-06-06 10:55:03,710 DEBUG awx.main.dispatch task bcc30a69-d00c-4065-9613-001f7d10b2ac starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:55:03,711 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:55:03,717 DEBUG awx.main.dispatch task bcede725-1d9e-405f-b69d-0c109fc8022c is finished
2021-06-06 10:55:03,727 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:55:03,756 DEBUG awx.main.dispatch task bcc30a69-d00c-4065-9613-001f7d10b2ac is finished
2021-06-06 10:55:23,706 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a632604a-d02d-49f5-bc83-65046a54f616, queue=awx_private_queue)
[2021-06-06 10:55:23,706: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a632604a-d02d-49f5-bc83-65046a54f616, queue=awx_private_queue)
2021-06-06 10:55:23,722 DEBUG awx.main.dispatch delivered a632604a-d02d-49f5-bc83-65046a54f616 to worker[183] qsize 0
2021-06-06 10:55:23,725 DEBUG awx.main.dispatch task a632604a-d02d-49f5-bc83-65046a54f616 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:55:23,727 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:55:23,748 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:55:23,776 DEBUG awx.main.dispatch task a632604a-d02d-49f5-bc83-65046a54f616 is finished
2021-06-06 10:55:33,657 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9649ec98-b4ae-44a5-8b5c-6ea1f82e94af, queue=awx_private_queue)
[2021-06-06 10:55:33,657: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9649ec98-b4ae-44a5-8b5c-6ea1f82e94af, queue=awx_private_queue)
2021-06-06 10:55:33,670 DEBUG awx.main.dispatch delivered 9649ec98-b4ae-44a5-8b5c-6ea1f82e94af to worker[183] qsize 0
2021-06-06 10:55:33,673 DEBUG awx.main.dispatch task 9649ec98-b4ae-44a5-8b5c-6ea1f82e94af starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:55:33,682 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:55:33,685 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:55:03.694687+00:00
2021-06-06 10:55:33,696 DEBUG awx.main.dispatch task 9649ec98-b4ae-44a5-8b5c-6ea1f82e94af is finished
2021-06-06 10:55:43,706 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c32eedcd-a22b-41ba-92fa-9fed0553f055, queue=awx_private_queue)
[2021-06-06 10:55:43,706: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c32eedcd-a22b-41ba-92fa-9fed0553f055, queue=awx_private_queue)
2021-06-06 10:55:43,725 DEBUG awx.main.dispatch delivered c32eedcd-a22b-41ba-92fa-9fed0553f055 to worker[183] qsize 0
2021-06-06 10:55:43,727 DEBUG awx.main.dispatch task c32eedcd-a22b-41ba-92fa-9fed0553f055 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:55:43,728 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:55:43,743 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:55:43,768 DEBUG awx.main.dispatch task c32eedcd-a22b-41ba-92fa-9fed0553f055 is finished
RESULT 2
OKREADY
2021-06-06 10:56:03,415 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c9526723-7b52-4505-9797-d08757eb16f7, queue=dev-awx-0)
[2021-06-06 10:56:03,415: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c9526723-7b52-4505-9797-d08757eb16f7, queue=dev-awx-0)
2021-06-06 10:56:03,657 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ffedb988-009c-41d7-a317-7ab4e6daed0c, queue=dev-awx-0)
[2021-06-06 10:56:03,657: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ffedb988-009c-41d7-a317-7ab4e6daed0c, queue=dev-awx-0)
2021-06-06 10:56:03,667 DEBUG awx.main.dispatch delivered c9526723-7b52-4505-9797-d08757eb16f7 to worker[183] qsize 0
2021-06-06 10:56:03,670 DEBUG awx.main.dispatch task c9526723-7b52-4505-9797-d08757eb16f7 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 10:56:03,671 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 10:56:03,675 DEBUG awx.main.dispatch delivered ffedb988-009c-41d7-a317-7ab4e6daed0c to worker[181] qsize 0
2021-06-06 10:56:03,677 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ce60f710-51b3-4fcd-95f0-3a35b9f599c0, queue=awx_private_queue)
[2021-06-06 10:56:03,677: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ce60f710-51b3-4fcd-95f0-3a35b9f599c0, queue=awx_private_queue)
2021-06-06 10:56:03,677 DEBUG awx.main.dispatch task ffedb988-009c-41d7-a317-7ab4e6daed0c starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 10:56:03,689 DEBUG awx.main.dispatch task ffedb988-009c-41d7-a317-7ab4e6daed0c is finished
2021-06-06 10:56:03,694 DEBUG awx.main.dispatch delivered ce60f710-51b3-4fcd-95f0-3a35b9f599c0 to worker[180] qsize 0
2021-06-06 10:56:03,697 DEBUG awx.main.dispatch task c9526723-7b52-4505-9797-d08757eb16f7 is finished
2021-06-06 10:56:03,696 DEBUG awx.main.dispatch task ce60f710-51b3-4fcd-95f0-3a35b9f599c0 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:56:03,706 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:56:03,706 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9659f279-724f-4ed8-aab5-59245e6d51f5, queue=awx_private_queue)
[2021-06-06 10:56:03,706: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9659f279-724f-4ed8-aab5-59245e6d51f5, queue=awx_private_queue)
2021-06-06 10:56:03,709 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:55:33.682842+00:00
2021-06-06 10:56:03,722 DEBUG awx.main.dispatch task ce60f710-51b3-4fcd-95f0-3a35b9f599c0 is finished
2021-06-06 10:56:03,723 DEBUG awx.main.dispatch delivered 9659f279-724f-4ed8-aab5-59245e6d51f5 to worker[182] qsize 0
2021-06-06 10:56:03,726 DEBUG awx.main.dispatch task 9659f279-724f-4ed8-aab5-59245e6d51f5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:56:03,730 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:56:03,748 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:56:03,775 DEBUG awx.main.dispatch task 9659f279-724f-4ed8-aab5-59245e6d51f5 is finished
2021-06-06 10:56:23,722 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bb61ef6d-8a82-4431-901b-c0c46e411246, queue=awx_private_queue)
[2021-06-06 10:56:23,722: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bb61ef6d-8a82-4431-901b-c0c46e411246, queue=awx_private_queue)
2021-06-06 10:56:23,748 DEBUG awx.main.dispatch delivered bb61ef6d-8a82-4431-901b-c0c46e411246 to worker[180] qsize 0
2021-06-06 10:56:23,755 DEBUG awx.main.dispatch task bb61ef6d-8a82-4431-901b-c0c46e411246 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:56:23,758 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:56:23,784 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:56:23,811 DEBUG awx.main.dispatch task bb61ef6d-8a82-4431-901b-c0c46e411246 is finished
2021-06-06 10:56:33,677 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1921a32f-05c9-4090-a743-1be11b4af583, queue=awx_private_queue)
[2021-06-06 10:56:33,677: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1921a32f-05c9-4090-a743-1be11b4af583, queue=awx_private_queue)
2021-06-06 10:56:33,693 DEBUG awx.main.dispatch delivered 1921a32f-05c9-4090-a743-1be11b4af583 to worker[182] qsize 0
2021-06-06 10:56:33,697 DEBUG awx.main.dispatch task 1921a32f-05c9-4090-a743-1be11b4af583 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:56:33,709 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:56:33,713 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:56:03.706972+00:00
2021-06-06 10:56:33,728 DEBUG awx.main.dispatch task 1921a32f-05c9-4090-a743-1be11b4af583 is finished
2021-06-06 10:56:43,717 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8d7a0ebe-eba6-42a3-904d-cc67eb9791c2, queue=awx_private_queue)
[2021-06-06 10:56:43,717: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8d7a0ebe-eba6-42a3-904d-cc67eb9791c2, queue=awx_private_queue)
2021-06-06 10:56:43,736 DEBUG awx.main.dispatch delivered 8d7a0ebe-eba6-42a3-904d-cc67eb9791c2 to worker[180] qsize 0
2021-06-06 10:56:43,739 DEBUG awx.main.dispatch task 8d7a0ebe-eba6-42a3-904d-cc67eb9791c2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:56:43,740 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:56:43,755 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:56:43,791 DEBUG awx.main.dispatch task 8d7a0ebe-eba6-42a3-904d-cc67eb9791c2 is finished
RESULT 2
OKREADY
2021-06-06 10:57:03,424 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c917fba3-420e-4962-9aca-ff76570e7180, queue=dev-awx-0)
[2021-06-06 10:57:03,424: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c917fba3-420e-4962-9aca-ff76570e7180, queue=dev-awx-0)
2021-06-06 10:57:03,669 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(85113c53-f343-4d59-9d53-4e6bde59115a, queue=dev-awx-0)
[2021-06-06 10:57:03,669: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(85113c53-f343-4d59-9d53-4e6bde59115a, queue=dev-awx-0)
2021-06-06 10:57:03,683 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(85652fdb-7627-4bb8-80ad-3374986f9b52, queue=awx_private_queue)
[2021-06-06 10:57:03,683: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(85652fdb-7627-4bb8-80ad-3374986f9b52, queue=awx_private_queue)
2021-06-06 10:57:03,688 DEBUG awx.main.dispatch delivered c917fba3-420e-4962-9aca-ff76570e7180 to worker[181] qsize 0
2021-06-06 10:57:03,691 DEBUG awx.main.dispatch delivered 85113c53-f343-4d59-9d53-4e6bde59115a to worker[180] qsize 0
2021-06-06 10:57:03,694 DEBUG awx.main.dispatch task 85113c53-f343-4d59-9d53-4e6bde59115a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 10:57:03,697 DEBUG awx.main.dispatch delivered 85652fdb-7627-4bb8-80ad-3374986f9b52 to worker[182] qsize 0
2021-06-06 10:57:03,694 DEBUG awx.main.dispatch task c917fba3-420e-4962-9aca-ff76570e7180 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 10:57:03,701 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 10:57:03,700 DEBUG awx.main.dispatch task 85652fdb-7627-4bb8-80ad-3374986f9b52 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:57:03,715 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:57:03,713 DEBUG awx.main.dispatch task 85113c53-f343-4d59-9d53-4e6bde59115a is finished
2021-06-06 10:57:03,717 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(facd6319-c4a1-4566-ab12-7e049787f132, queue=awx_private_queue)
[2021-06-06 10:57:03,717: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(facd6319-c4a1-4566-ab12-7e049787f132, queue=awx_private_queue)
2021-06-06 10:57:03,722 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:56:33.709255+00:00
2021-06-06 10:57:03,733 DEBUG awx.main.dispatch task c917fba3-420e-4962-9aca-ff76570e7180 is finished
2021-06-06 10:57:03,735 DEBUG awx.main.dispatch delivered facd6319-c4a1-4566-ab12-7e049787f132 to worker[183] qsize 0
2021-06-06 10:57:03,736 DEBUG awx.main.dispatch task 85652fdb-7627-4bb8-80ad-3374986f9b52 is finished
2021-06-06 10:57:03,739 DEBUG awx.main.dispatch task facd6319-c4a1-4566-ab12-7e049787f132 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:57:03,745 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:57:03,771 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:57:03,805 DEBUG awx.main.dispatch task facd6319-c4a1-4566-ab12-7e049787f132 is finished
2021-06-06 10:57:23,731 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7f749f7a-3544-4b9f-835d-c2daede6a31e, queue=awx_private_queue)
[2021-06-06 10:57:23,731: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7f749f7a-3544-4b9f-835d-c2daede6a31e, queue=awx_private_queue)
2021-06-06 10:57:23,766 DEBUG awx.main.dispatch delivered 7f749f7a-3544-4b9f-835d-c2daede6a31e to worker[180] qsize 0
2021-06-06 10:57:23,772 DEBUG awx.main.dispatch task 7f749f7a-3544-4b9f-835d-c2daede6a31e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:57:23,776 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:57:23,796 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:57:23,824 DEBUG awx.main.dispatch task 7f749f7a-3544-4b9f-835d-c2daede6a31e is finished
2021-06-06 10:57:33,684 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(14d5c817-5dd3-45f1-808c-d8e6e87984c3, queue=awx_private_queue)
[2021-06-06 10:57:33,684: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(14d5c817-5dd3-45f1-808c-d8e6e87984c3, queue=awx_private_queue)
2021-06-06 10:57:33,700 DEBUG awx.main.dispatch delivered 14d5c817-5dd3-45f1-808c-d8e6e87984c3 to worker[180] qsize 0
2021-06-06 10:57:33,703 DEBUG awx.main.dispatch task 14d5c817-5dd3-45f1-808c-d8e6e87984c3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:57:33,711 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:57:33,714 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:57:03.715860+00:00
2021-06-06 10:57:33,730 DEBUG awx.main.dispatch task 14d5c817-5dd3-45f1-808c-d8e6e87984c3 is finished
2021-06-06 10:57:43,728 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(307f40ee-4357-4dfb-a0a9-7765873ea39f, queue=awx_private_queue)
[2021-06-06 10:57:43,728: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(307f40ee-4357-4dfb-a0a9-7765873ea39f, queue=awx_private_queue)
2021-06-06 10:57:43,747 DEBUG awx.main.dispatch delivered 307f40ee-4357-4dfb-a0a9-7765873ea39f to worker[181] qsize 0
2021-06-06 10:57:43,750 DEBUG awx.main.dispatch task 307f40ee-4357-4dfb-a0a9-7765873ea39f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:57:43,752 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:57:43,772 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:57:43,804 DEBUG awx.main.dispatch task 307f40ee-4357-4dfb-a0a9-7765873ea39f is finished
RESULT 2
OKREADY
2021-06-06 10:58:03,430 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(abf7eefe-2101-4065-8bef-5c45f811e35f, queue=dev-awx-0)
[2021-06-06 10:58:03,430: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(abf7eefe-2101-4065-8bef-5c45f811e35f, queue=dev-awx-0)
2021-06-06 10:58:03,669 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(60f1f615-49fe-412a-947e-730acf122b66, queue=dev-awx-0)
[2021-06-06 10:58:03,669: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(60f1f615-49fe-412a-947e-730acf122b66, queue=dev-awx-0)
2021-06-06 10:58:03,677 DEBUG awx.main.dispatch delivered abf7eefe-2101-4065-8bef-5c45f811e35f to worker[181] qsize 0
2021-06-06 10:58:03,679 DEBUG awx.main.dispatch task abf7eefe-2101-4065-8bef-5c45f811e35f starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 10:58:03,681 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 10:58:03,686 DEBUG awx.main.dispatch delivered 60f1f615-49fe-412a-947e-730acf122b66 to worker[180] qsize 0
2021-06-06 10:58:03,687 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a5700229-191e-4403-9dfb-c978ade4b0e1, queue=awx_private_queue)
[2021-06-06 10:58:03,687: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a5700229-191e-4403-9dfb-c978ade4b0e1, queue=awx_private_queue)
2021-06-06 10:58:03,688 DEBUG awx.main.dispatch task 60f1f615-49fe-412a-947e-730acf122b66 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 10:58:03,699 DEBUG awx.main.dispatch task 60f1f615-49fe-412a-947e-730acf122b66 is finished
2021-06-06 10:58:03,703 DEBUG awx.main.dispatch delivered a5700229-191e-4403-9dfb-c978ade4b0e1 to worker[183] qsize 0
2021-06-06 10:58:03,703 DEBUG awx.main.dispatch task abf7eefe-2101-4065-8bef-5c45f811e35f is finished
2021-06-06 10:58:03,705 DEBUG awx.main.dispatch task a5700229-191e-4403-9dfb-c978ade4b0e1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:58:03,716 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:58:03,721 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:57:33.712035+00:00
2021-06-06 10:58:03,727 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f06cdbc1-8e01-446a-bf3d-dc568466e1d8, queue=awx_private_queue)
[2021-06-06 10:58:03,727: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f06cdbc1-8e01-446a-bf3d-dc568466e1d8, queue=awx_private_queue)
2021-06-06 10:58:03,734 DEBUG awx.main.dispatch task a5700229-191e-4403-9dfb-c978ade4b0e1 is finished
2021-06-06 10:58:03,742 DEBUG awx.main.dispatch delivered f06cdbc1-8e01-446a-bf3d-dc568466e1d8 to worker[182] qsize 0
2021-06-06 10:58:03,744 DEBUG awx.main.dispatch task f06cdbc1-8e01-446a-bf3d-dc568466e1d8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:58:03,746 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:58:03,763 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:58:03,789 DEBUG awx.main.dispatch task f06cdbc1-8e01-446a-bf3d-dc568466e1d8 is finished
2021-06-06 10:58:23,736 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0dc85865-79f9-41d2-a3b8-792bbb318a70, queue=awx_private_queue)
[2021-06-06 10:58:23,736: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0dc85865-79f9-41d2-a3b8-792bbb318a70, queue=awx_private_queue)
2021-06-06 10:58:23,754 DEBUG awx.main.dispatch delivered 0dc85865-79f9-41d2-a3b8-792bbb318a70 to worker[183] qsize 0
2021-06-06 10:58:23,756 DEBUG awx.main.dispatch task 0dc85865-79f9-41d2-a3b8-792bbb318a70 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:58:23,760 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:58:23,779 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:58:23,814 DEBUG awx.main.dispatch task 0dc85865-79f9-41d2-a3b8-792bbb318a70 is finished
2021-06-06 10:58:33,688 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6c828be1-8ae7-4aa0-9370-185cb3839e08, queue=awx_private_queue)
[2021-06-06 10:58:33,688: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6c828be1-8ae7-4aa0-9370-185cb3839e08, queue=awx_private_queue)
2021-06-06 10:58:33,703 DEBUG awx.main.dispatch delivered 6c828be1-8ae7-4aa0-9370-185cb3839e08 to worker[180] qsize 0
2021-06-06 10:58:33,707 DEBUG awx.main.dispatch task 6c828be1-8ae7-4aa0-9370-185cb3839e08 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:58:33,717 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:58:33,721 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:58:03.717060+00:00
2021-06-06 10:58:33,732 DEBUG awx.main.dispatch task 6c828be1-8ae7-4aa0-9370-185cb3839e08 is finished
2021-06-06 10:58:43,736 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(98ac0e21-0749-426e-95ab-650e82558082, queue=awx_private_queue)
[2021-06-06 10:58:43,736: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(98ac0e21-0749-426e-95ab-650e82558082, queue=awx_private_queue)
2021-06-06 10:58:43,751 DEBUG awx.main.dispatch delivered 98ac0e21-0749-426e-95ab-650e82558082 to worker[183] qsize 0
2021-06-06 10:58:43,754 DEBUG awx.main.dispatch task 98ac0e21-0749-426e-95ab-650e82558082 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:58:43,755 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:58:43,773 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:58:43,799 DEBUG awx.main.dispatch task 98ac0e21-0749-426e-95ab-650e82558082 is finished
RESULT 2
OKREADY
2021-06-06 10:59:03,385 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(93e54efb-aaca-4455-8e0b-bc95b00c5131, queue=dev-awx-0)
[2021-06-06 10:59:03,385: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(93e54efb-aaca-4455-8e0b-bc95b00c5131, queue=dev-awx-0)
2021-06-06 10:59:03,439 DEBUG awx.main.dispatch delivered 93e54efb-aaca-4455-8e0b-bc95b00c5131 to worker[183] qsize 0
2021-06-06 10:59:03,446 DEBUG awx.main.dispatch task 93e54efb-aaca-4455-8e0b-bc95b00c5131 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-06-06 10:59:03,450 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-06-06 10:59:03,471 DEBUG awx.main.dispatch task 93e54efb-aaca-4455-8e0b-bc95b00c5131 is finished
2021-06-06 10:59:03,437 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(4654b3dd-c82b-4680-ad86-6b44abbe814e, queue=dev-awx-0)
[2021-06-06 10:59:03,437: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(4654b3dd-c82b-4680-ad86-6b44abbe814e, queue=dev-awx-0)
2021-06-06 10:59:03,667 DEBUG awx.main.dispatch delivered 4654b3dd-c82b-4680-ad86-6b44abbe814e to worker[181] qsize 0
2021-06-06 10:59:03,669 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(44fd1ccb-ab65-48a4-ad4c-244ce441265f, queue=dev-awx-0)
[2021-06-06 10:59:03,669: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(44fd1ccb-ab65-48a4-ad4c-244ce441265f, queue=dev-awx-0)
2021-06-06 10:59:03,670 DEBUG awx.main.dispatch task 4654b3dd-c82b-4680-ad86-6b44abbe814e starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 10:59:03,673 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 10:59:03,684 DEBUG awx.main.dispatch delivered 44fd1ccb-ab65-48a4-ad4c-244ce441265f to worker[180] qsize 0
2021-06-06 10:59:03,687 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6ff4f772-3cc8-462b-ae52-e771315ac1b4, queue=awx_private_queue)
[2021-06-06 10:59:03,687: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6ff4f772-3cc8-462b-ae52-e771315ac1b4, queue=awx_private_queue)
2021-06-06 10:59:03,687 DEBUG awx.main.dispatch task 44fd1ccb-ab65-48a4-ad4c-244ce441265f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 10:59:03,699 DEBUG awx.main.dispatch task 4654b3dd-c82b-4680-ad86-6b44abbe814e is finished
2021-06-06 10:59:03,703 DEBUG awx.main.dispatch task 44fd1ccb-ab65-48a4-ad4c-244ce441265f is finished
2021-06-06 10:59:03,708 DEBUG awx.main.dispatch delivered 6ff4f772-3cc8-462b-ae52-e771315ac1b4 to worker[182] qsize 0
2021-06-06 10:59:03,712 DEBUG awx.main.dispatch task 6ff4f772-3cc8-462b-ae52-e771315ac1b4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:59:03,724 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:59:03,728 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:58:33.717777+00:00
2021-06-06 10:59:03,737 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bf4c2e60-dc21-441e-8b5d-67d7b823a864, queue=awx_private_queue)
[2021-06-06 10:59:03,737: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bf4c2e60-dc21-441e-8b5d-67d7b823a864, queue=awx_private_queue)
2021-06-06 10:59:03,741 DEBUG awx.main.dispatch task 6ff4f772-3cc8-462b-ae52-e771315ac1b4 is finished
2021-06-06 10:59:03,754 DEBUG awx.main.dispatch delivered bf4c2e60-dc21-441e-8b5d-67d7b823a864 to worker[181] qsize 0
2021-06-06 10:59:03,756 DEBUG awx.main.dispatch task bf4c2e60-dc21-441e-8b5d-67d7b823a864 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:59:03,758 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:59:03,773 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:59:03,803 DEBUG awx.main.dispatch task bf4c2e60-dc21-441e-8b5d-67d7b823a864 is finished
2021-06-06 10:59:23,748 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(89304bc9-f4fc-4543-91db-a3a2b6d50f40, queue=awx_private_queue)
[2021-06-06 10:59:23,748: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(89304bc9-f4fc-4543-91db-a3a2b6d50f40, queue=awx_private_queue)
2021-06-06 10:59:23,764 DEBUG awx.main.dispatch delivered 89304bc9-f4fc-4543-91db-a3a2b6d50f40 to worker[180] qsize 0
2021-06-06 10:59:23,766 DEBUG awx.main.dispatch task 89304bc9-f4fc-4543-91db-a3a2b6d50f40 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:59:23,767 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:59:23,783 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:59:23,817 DEBUG awx.main.dispatch task 89304bc9-f4fc-4543-91db-a3a2b6d50f40 is finished
2021-06-06 10:59:33,688 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(00c9ee54-fafc-4fa5-9378-1ecbdbd7b05f, queue=awx_private_queue)
[2021-06-06 10:59:33,688: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(00c9ee54-fafc-4fa5-9378-1ecbdbd7b05f, queue=awx_private_queue)
2021-06-06 10:59:33,702 DEBUG awx.main.dispatch delivered 00c9ee54-fafc-4fa5-9378-1ecbdbd7b05f to worker[180] qsize 0
2021-06-06 10:59:33,705 DEBUG awx.main.dispatch task 00c9ee54-fafc-4fa5-9378-1ecbdbd7b05f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 10:59:33,715 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 10:59:33,718 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:59:03.724399+00:00
2021-06-06 10:59:33,737 DEBUG awx.main.dispatch task 00c9ee54-fafc-4fa5-9378-1ecbdbd7b05f is finished
2021-06-06 10:59:43,747 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(64ab61e6-8649-4a1b-8bb1-a7115a705c6a, queue=awx_private_queue)
[2021-06-06 10:59:43,747: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(64ab61e6-8649-4a1b-8bb1-a7115a705c6a, queue=awx_private_queue)
2021-06-06 10:59:43,764 DEBUG awx.main.dispatch delivered 64ab61e6-8649-4a1b-8bb1-a7115a705c6a to worker[183] qsize 0
2021-06-06 10:59:43,767 DEBUG awx.main.dispatch task 64ab61e6-8649-4a1b-8bb1-a7115a705c6a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 10:59:43,769 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 10:59:43,791 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 10:59:43,825 DEBUG awx.main.dispatch task 64ab61e6-8649-4a1b-8bb1-a7115a705c6a is finished
RESULT 2
OKREADY
2021-06-06 11:00:03,447 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(1d7221b8-e9e5-483f-a606-6fb8f3187e03, queue=dev-awx-0)
[2021-06-06 11:00:03,447: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(1d7221b8-e9e5-483f-a606-6fb8f3187e03, queue=dev-awx-0)
2021-06-06 11:00:03,694 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(38286cdf-2987-49a6-b84b-98abd5af1348, queue=dev-awx-0)
[2021-06-06 11:00:03,694: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(38286cdf-2987-49a6-b84b-98abd5af1348, queue=dev-awx-0)
2021-06-06 11:00:03,735 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b70e2558-4322-4a29-b4af-d676a4b0a657, queue=awx_private_queue)
[2021-06-06 11:00:03,735: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b70e2558-4322-4a29-b4af-d676a4b0a657, queue=awx_private_queue)
2021-06-06 11:00:03,756 DEBUG awx.main.dispatch delivered 1d7221b8-e9e5-483f-a606-6fb8f3187e03 to worker[182] qsize 0
2021-06-06 11:00:03,757 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(eb2ff422-4298-452f-8fd6-3776ead904c6, queue=awx_private_queue)
[2021-06-06 11:00:03,757: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(eb2ff422-4298-452f-8fd6-3776ead904c6, queue=awx_private_queue)
2021-06-06 11:00:03,758 DEBUG awx.main.dispatch task 1d7221b8-e9e5-483f-a606-6fb8f3187e03 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 11:00:03,760 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 11:00:03,763 DEBUG awx.main.dispatch delivered 38286cdf-2987-49a6-b84b-98abd5af1348 to worker[181] qsize 0
2021-06-06 11:00:03,765 DEBUG awx.main.dispatch delivered b70e2558-4322-4a29-b4af-d676a4b0a657 to worker[180] qsize 0
2021-06-06 11:00:03,765 DEBUG awx.main.dispatch task 38286cdf-2987-49a6-b84b-98abd5af1348 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:00:03,767 DEBUG awx.main.dispatch task b70e2558-4322-4a29-b4af-d676a4b0a657 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:00:03,772 DEBUG awx.main.dispatch delivered eb2ff422-4298-452f-8fd6-3776ead904c6 to worker[183] qsize 0
2021-06-06 11:00:03,773 DEBUG awx.main.dispatch task eb2ff422-4298-452f-8fd6-3776ead904c6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:00:03,775 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:00:03,821 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:00:03,830 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 10:59:33.715344+00:00
2021-06-06 11:00:03,831 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 11:00:03,830 DEBUG awx.main.dispatch task 38286cdf-2987-49a6-b84b-98abd5af1348 is finished
2021-06-06 11:00:03,844 DEBUG awx.main.dispatch task 1d7221b8-e9e5-483f-a606-6fb8f3187e03 is finished
2021-06-06 11:00:03,858 DEBUG awx.main.dispatch task b70e2558-4322-4a29-b4af-d676a4b0a657 is finished
2021-06-06 11:00:03,913 DEBUG awx.main.dispatch task eb2ff422-4298-452f-8fd6-3776ead904c6 is finished
2021-06-06 11:00:23,769 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7e6ad720-f3d2-47ce-a32b-fd79fb282bff, queue=awx_private_queue)
[2021-06-06 11:00:23,769: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7e6ad720-f3d2-47ce-a32b-fd79fb282bff, queue=awx_private_queue)
2021-06-06 11:00:23,785 DEBUG awx.main.dispatch delivered 7e6ad720-f3d2-47ce-a32b-fd79fb282bff to worker[180] qsize 0
2021-06-06 11:00:23,787 DEBUG awx.main.dispatch task 7e6ad720-f3d2-47ce-a32b-fd79fb282bff starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:00:23,789 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:00:23,807 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 11:00:23,848 DEBUG awx.main.dispatch task 7e6ad720-f3d2-47ce-a32b-fd79fb282bff is finished
2021-06-06 11:00:33,737 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7c86201a-9d58-4447-9ace-ffda4959d3d8, queue=awx_private_queue)
[2021-06-06 11:00:33,737: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7c86201a-9d58-4447-9ace-ffda4959d3d8, queue=awx_private_queue)
2021-06-06 11:00:33,772 DEBUG awx.main.dispatch delivered 7c86201a-9d58-4447-9ace-ffda4959d3d8 to worker[183] qsize 0
2021-06-06 11:00:33,780 DEBUG awx.main.dispatch task 7c86201a-9d58-4447-9ace-ffda4959d3d8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:00:33,823 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:00:34,254 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 11:00:03.821997+00:00
2021-06-06 11:00:34,369 DEBUG awx.main.dispatch task 7c86201a-9d58-4447-9ace-ffda4959d3d8 is finished
2021-06-06 11:00:43,772 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6823de55-93c8-4723-95aa-99803a6dbbbd, queue=awx_private_queue)
[2021-06-06 11:00:43,772: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6823de55-93c8-4723-95aa-99803a6dbbbd, queue=awx_private_queue)
2021-06-06 11:00:43,803 DEBUG awx.main.dispatch delivered 6823de55-93c8-4723-95aa-99803a6dbbbd to worker[182] qsize 0
2021-06-06 11:00:43,808 DEBUG awx.main.dispatch task 6823de55-93c8-4723-95aa-99803a6dbbbd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:00:43,812 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:00:43,843 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 11:00:43,888 DEBUG awx.main.dispatch task 6823de55-93c8-4723-95aa-99803a6dbbbd is finished
RESULT 2
OKREADY
2021-06-06 11:01:03,447 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a0cb9cad-6268-47fd-a07f-b885fd665537, queue=dev-awx-0)
[2021-06-06 11:01:03,447: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a0cb9cad-6268-47fd-a07f-b885fd665537, queue=dev-awx-0)
2021-06-06 11:01:03,524 DEBUG awx.main.dispatch delivered a0cb9cad-6268-47fd-a07f-b885fd665537 to worker[181] qsize 0
2021-06-06 11:01:03,528 DEBUG awx.main.dispatch task a0cb9cad-6268-47fd-a07f-b885fd665537 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 11:01:03,530 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 11:01:03,575 DEBUG awx.main.dispatch task a0cb9cad-6268-47fd-a07f-b885fd665537 is finished
2021-06-06 11:01:03,695 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a24f2f58-ac11-44c3-bc16-649d67255cad, queue=dev-awx-0)
[2021-06-06 11:01:03,695: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a24f2f58-ac11-44c3-bc16-649d67255cad, queue=dev-awx-0)
2021-06-06 11:01:04,024 DEBUG awx.main.dispatch delivered a24f2f58-ac11-44c3-bc16-649d67255cad to worker[182] qsize 0
2021-06-06 11:01:04,025 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7d1c3fa2-ff46-4795-98e7-6e539cfc73ec, queue=awx_private_queue)
[2021-06-06 11:01:04,025: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7d1c3fa2-ff46-4795-98e7-6e539cfc73ec, queue=awx_private_queue)
2021-06-06 11:01:04,027 DEBUG awx.main.dispatch task a24f2f58-ac11-44c3-bc16-649d67255cad starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:01:04,040 DEBUG awx.main.dispatch delivered 7d1c3fa2-ff46-4795-98e7-6e539cfc73ec to worker[181] qsize 0
2021-06-06 11:01:04,041 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b1537c95-5bb1-4371-8f4b-6f699e9f2f9e, queue=awx_private_queue)
[2021-06-06 11:01:04,041: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b1537c95-5bb1-4371-8f4b-6f699e9f2f9e, queue=awx_private_queue)
2021-06-06 11:01:04,043 DEBUG awx.main.dispatch task 7d1c3fa2-ff46-4795-98e7-6e539cfc73ec starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:01:04,058 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:01:04,065 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 11:00:33.823298+00:00
2021-06-06 11:01:04,068 DEBUG awx.main.dispatch delivered b1537c95-5bb1-4371-8f4b-6f699e9f2f9e to worker[180] qsize 0
2021-06-06 11:01:04,071 DEBUG awx.main.dispatch task a24f2f58-ac11-44c3-bc16-649d67255cad is finished
2021-06-06 11:01:04,072 DEBUG awx.main.dispatch task b1537c95-5bb1-4371-8f4b-6f699e9f2f9e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:01:04,076 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:01:04,092 DEBUG awx.main.dispatch task 7d1c3fa2-ff46-4795-98e7-6e539cfc73ec is finished
2021-06-06 11:01:04,113 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 11:01:04,159 DEBUG awx.main.dispatch task b1537c95-5bb1-4371-8f4b-6f699e9f2f9e is finished
2021-06-06 11:01:24,054 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f18a31a9-ebb8-45c4-87b2-686ab32f405f, queue=awx_private_queue)
[2021-06-06 11:01:24,054: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f18a31a9-ebb8-45c4-87b2-686ab32f405f, queue=awx_private_queue)
2021-06-06 11:01:24,074 DEBUG awx.main.dispatch delivered f18a31a9-ebb8-45c4-87b2-686ab32f405f to worker[182] qsize 0
2021-06-06 11:01:24,078 DEBUG awx.main.dispatch task f18a31a9-ebb8-45c4-87b2-686ab32f405f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:01:24,081 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:01:24,127 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 11:01:24,174 DEBUG awx.main.dispatch task f18a31a9-ebb8-45c4-87b2-686ab32f405f is finished
2021-06-06 11:01:34,033 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6506cb8d-1b10-4ae9-9c1d-d150c4022aa7, queue=awx_private_queue)
[2021-06-06 11:01:34,033: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6506cb8d-1b10-4ae9-9c1d-d150c4022aa7, queue=awx_private_queue)
2021-06-06 11:01:34,069 DEBUG awx.main.dispatch delivered 6506cb8d-1b10-4ae9-9c1d-d150c4022aa7 to worker[183] qsize 0
2021-06-06 11:01:34,079 DEBUG awx.main.dispatch task 6506cb8d-1b10-4ae9-9c1d-d150c4022aa7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:01:34,107 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:01:34,362 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 11:01:04.059034+00:00
2021-06-06 11:01:34,649 DEBUG awx.main.dispatch task 6506cb8d-1b10-4ae9-9c1d-d150c4022aa7 is finished
2021-06-06 11:01:44,054 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(698f00b2-e5a8-44a6-b630-0a0a7a58eac0, queue=awx_private_queue)
[2021-06-06 11:01:44,054: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(698f00b2-e5a8-44a6-b630-0a0a7a58eac0, queue=awx_private_queue)
2021-06-06 11:01:44,078 DEBUG awx.main.dispatch delivered 698f00b2-e5a8-44a6-b630-0a0a7a58eac0 to worker[180] qsize 0
2021-06-06 11:01:44,084 DEBUG awx.main.dispatch task 698f00b2-e5a8-44a6-b630-0a0a7a58eac0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:01:44,090 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:01:44,153 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 11:01:44,224 DEBUG awx.main.dispatch task 698f00b2-e5a8-44a6-b630-0a0a7a58eac0 is finished
RESULT 2
OKREADY
2021-06-06 11:02:03,457 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(0fe49062-1d8b-42c9-b833-1051aea9cde0, queue=dev-awx-0)
[2021-06-06 11:02:03,457: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(0fe49062-1d8b-42c9-b833-1051aea9cde0, queue=dev-awx-0)
2021-06-06 11:02:04,409 DEBUG awx.main.dispatch delivered 0fe49062-1d8b-42c9-b833-1051aea9cde0 to worker[182] qsize 0
2021-06-06 11:02:04,393 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(60c360ca-962b-4b61-8d83-4967cc8c979c, queue=dev-awx-0)
[2021-06-06 11:02:04,393: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(60c360ca-962b-4b61-8d83-4967cc8c979c, queue=dev-awx-0)
2021-06-06 11:02:04,637 DEBUG awx.main.dispatch delivered 60c360ca-962b-4b61-8d83-4967cc8c979c to worker[183] qsize 0
2021-06-06 11:02:04,638 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4b1a6a56-813c-4f9a-9384-5223235798ca, queue=awx_private_queue)
[2021-06-06 11:02:04,638: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4b1a6a56-813c-4f9a-9384-5223235798ca, queue=awx_private_queue)
2021-06-06 11:02:04,639 DEBUG awx.main.dispatch task 60c360ca-962b-4b61-8d83-4967cc8c979c starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:02:04,412 DEBUG awx.main.dispatch task 0fe49062-1d8b-42c9-b833-1051aea9cde0 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 11:02:04,643 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 11:02:04,654 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(44de1d14-482a-45db-bffe-e9cab26f6fd6, queue=awx_private_queue)
[2021-06-06 11:02:04,654: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(44de1d14-482a-45db-bffe-e9cab26f6fd6, queue=awx_private_queue)
2021-06-06 11:02:04,657 DEBUG awx.main.dispatch delivered 4b1a6a56-813c-4f9a-9384-5223235798ca to worker[181] qsize 0
2021-06-06 11:02:04,661 DEBUG awx.main.dispatch task 4b1a6a56-813c-4f9a-9384-5223235798ca starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:02:04,672 DEBUG awx.main.dispatch task 0fe49062-1d8b-42c9-b833-1051aea9cde0 is finished
2021-06-06 11:02:04,672 DEBUG awx.main.dispatch task 60c360ca-962b-4b61-8d83-4967cc8c979c is finished
2021-06-06 11:02:04,678 DEBUG awx.main.dispatch delivered 44de1d14-482a-45db-bffe-e9cab26f6fd6 to worker[183] qsize 0
2021-06-06 11:02:04,680 DEBUG awx.main.dispatch task 44de1d14-482a-45db-bffe-e9cab26f6fd6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:02:04,682 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:02:04,686 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:02:04,695 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 11:01:34.107322+00:00
2021-06-06 11:02:04,898 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 11:02:05,074 DEBUG awx.main.dispatch task 4b1a6a56-813c-4f9a-9384-5223235798ca is finished
2021-06-06 11:02:05,125 DEBUG awx.main.dispatch task 44de1d14-482a-45db-bffe-e9cab26f6fd6 is finished
2021-06-06 11:02:24,665 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a8e2ab4f-7ce5-4447-9077-d31ccbf76c7c, queue=awx_private_queue)
[2021-06-06 11:02:24,665: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a8e2ab4f-7ce5-4447-9077-d31ccbf76c7c, queue=awx_private_queue)
2021-06-06 11:02:24,680 DEBUG awx.main.dispatch delivered a8e2ab4f-7ce5-4447-9077-d31ccbf76c7c to worker[180] qsize 0
2021-06-06 11:02:24,683 DEBUG awx.main.dispatch task a8e2ab4f-7ce5-4447-9077-d31ccbf76c7c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:02:24,685 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:02:24,795 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 11:02:24,892 DEBUG awx.main.dispatch task a8e2ab4f-7ce5-4447-9077-d31ccbf76c7c is finished
2021-06-06 11:02:34,640 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6bba68d0-37d8-45fa-ba93-eec6f0fe9ecd, queue=awx_private_queue)
[2021-06-06 11:02:34,640: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6bba68d0-37d8-45fa-ba93-eec6f0fe9ecd, queue=awx_private_queue)
2021-06-06 11:02:34,665 DEBUG awx.main.dispatch delivered 6bba68d0-37d8-45fa-ba93-eec6f0fe9ecd to worker[181] qsize 0
2021-06-06 11:02:34,673 DEBUG awx.main.dispatch task 6bba68d0-37d8-45fa-ba93-eec6f0fe9ecd starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:02:35,005 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:02:35,010 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 11:02:04.686672+00:00
2021-06-06 11:02:35,116 DEBUG awx.main.dispatch task 6bba68d0-37d8-45fa-ba93-eec6f0fe9ecd is finished
2021-06-06 11:02:44,668 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2ca6e1d0-c4fd-4b6f-84af-507d184412bc, queue=awx_private_queue)
[2021-06-06 11:02:44,668: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2ca6e1d0-c4fd-4b6f-84af-507d184412bc, queue=awx_private_queue)
2021-06-06 11:02:44,692 DEBUG awx.main.dispatch delivered 2ca6e1d0-c4fd-4b6f-84af-507d184412bc to worker[180] qsize 0
2021-06-06 11:02:44,699 DEBUG awx.main.dispatch task 2ca6e1d0-c4fd-4b6f-84af-507d184412bc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:02:44,703 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:02:45,170 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 11:02:45,210 DEBUG awx.main.dispatch task 2ca6e1d0-c4fd-4b6f-84af-507d184412bc is finished
RESULT 2
OKREADY
2021-06-06 11:03:03,459 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6e2058dd-882e-4f4f-930a-a5cbd6cb6e40, queue=dev-awx-0)
[2021-06-06 11:03:03,459: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6e2058dd-882e-4f4f-930a-a5cbd6cb6e40, queue=dev-awx-0)
2021-06-06 11:03:03,877 DEBUG awx.main.dispatch delivered 6e2058dd-882e-4f4f-930a-a5cbd6cb6e40 to worker[180] qsize 0
2021-06-06 11:03:03,881 DEBUG awx.main.dispatch task 6e2058dd-882e-4f4f-930a-a5cbd6cb6e40 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 11:03:03,883 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 11:03:04,024 DEBUG awx.main.dispatch task 6e2058dd-882e-4f4f-930a-a5cbd6cb6e40 is finished
2021-06-06 11:03:04,392 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ab21ac21-28ec-4f66-8cb1-2065568a59ab, queue=dev-awx-0)
[2021-06-06 11:03:04,392: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ab21ac21-28ec-4f66-8cb1-2065568a59ab, queue=dev-awx-0)
2021-06-06 11:03:04,411 DEBUG awx.main.dispatch delivered ab21ac21-28ec-4f66-8cb1-2065568a59ab to worker[180] qsize 0
2021-06-06 11:03:04,414 DEBUG awx.main.dispatch task ab21ac21-28ec-4f66-8cb1-2065568a59ab starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:03:05,404 DEBUG awx.main.dispatch task ab21ac21-28ec-4f66-8cb1-2065568a59ab is finished
2021-06-06 11:03:04,639 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8516ff07-3043-4dd2-8137-1f16897ba9c1, queue=awx_private_queue)
[2021-06-06 11:03:04,639: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8516ff07-3043-4dd2-8137-1f16897ba9c1, queue=awx_private_queue)
2021-06-06 11:03:05,600 DEBUG awx.main.dispatch delivered 8516ff07-3043-4dd2-8137-1f16897ba9c1 to worker[182] qsize 0
2021-06-06 11:03:05,602 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2ab08ec6-afb4-401f-a4da-5da9a0be9cab, queue=awx_private_queue)
[2021-06-06 11:03:05,602: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2ab08ec6-afb4-401f-a4da-5da9a0be9cab, queue=awx_private_queue)
2021-06-06 11:03:05,603 DEBUG awx.main.dispatch task 8516ff07-3043-4dd2-8137-1f16897ba9c1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:03:05,625 DEBUG awx.main.dispatch delivered 2ab08ec6-afb4-401f-a4da-5da9a0be9cab to worker[181] qsize 0
2021-06-06 11:03:05,630 DEBUG awx.main.dispatch task 2ab08ec6-afb4-401f-a4da-5da9a0be9cab starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:03:05,636 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:03:05,842 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:03:05,847 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 11:02:35.005815+00:00
2021-06-06 11:03:05,849 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 11:03:05,879 DEBUG awx.main.dispatch task 8516ff07-3043-4dd2-8137-1f16897ba9c1 is finished
2021-06-06 11:03:05,892 DEBUG awx.main.dispatch task 2ab08ec6-afb4-401f-a4da-5da9a0be9cab is finished
2021-06-06 11:03:25,610 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b9199ce2-e3af-461f-b825-821054fefd2c, queue=awx_private_queue)
[2021-06-06 11:03:25,610: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b9199ce2-e3af-461f-b825-821054fefd2c, queue=awx_private_queue)
2021-06-06 11:03:25,632 DEBUG awx.main.dispatch delivered b9199ce2-e3af-461f-b825-821054fefd2c to worker[182] qsize 0
2021-06-06 11:03:25,635 DEBUG awx.main.dispatch task b9199ce2-e3af-461f-b825-821054fefd2c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:03:25,637 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:03:25,706 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 11:03:26,125 DEBUG awx.main.dispatch task b9199ce2-e3af-461f-b825-821054fefd2c is finished
2021-06-06 11:03:34,639 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ec184c09-9c99-43a4-b00a-ea392e7f40c6, queue=awx_private_queue)
[2021-06-06 11:03:34,639: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ec184c09-9c99-43a4-b00a-ea392e7f40c6, queue=awx_private_queue)
2021-06-06 11:03:34,660 DEBUG awx.main.dispatch delivered ec184c09-9c99-43a4-b00a-ea392e7f40c6 to worker[183] qsize 0
2021-06-06 11:03:34,665 DEBUG awx.main.dispatch task ec184c09-9c99-43a4-b00a-ea392e7f40c6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:03:34,717 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:03:34,725 DEBUG awx.main.tasks Last scheduler run was: 2021-06-06 11:03:05.842693+00:00
2021-06-06 11:03:34,770 DEBUG awx.main.dispatch task ec184c09-9c99-43a4-b00a-ea392e7f40c6 is finished
2021-06-06 11:03:45,612 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cd5fae11-5045-4e79-9c7d-8f81ed5a3197, queue=awx_private_queue)
[2021-06-06 11:03:45,612: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cd5fae11-5045-4e79-9c7d-8f81ed5a3197, queue=awx_private_queue)
2021-06-06 11:03:45,635 DEBUG awx.main.dispatch delivered cd5fae11-5045-4e79-9c7d-8f81ed5a3197 to worker[180] qsize 0
2021-06-06 11:03:45,639 DEBUG awx.main.dispatch task cd5fae11-5045-4e79-9c7d-8f81ed5a3197 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:03:45,643 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:03:45,932 DEBUG awx.main.scheduler Starting Scheduler
2021-06-06 11:03:45,976 DEBUG awx.main.dispatch task cd5fae11-5045-4e79-9c7d-8f81ed5a3197 is finished
RESULT 2
OKREADY
2021-06-06 11:04:03,465 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(234726d5-8096-4154-8eaf-1c0e83332891, queue=dev-awx-0)
[2021-06-06 11:04:03,465: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(234726d5-8096-4154-8eaf-1c0e83332891, queue=dev-awx-0)
2021-06-06 11:04:04,392 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(6f1b8f30-cf1c-45f2-8341-aef5e9d6d746, queue=dev-awx-0)
[2021-06-06 11:04:04,392: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(6f1b8f30-cf1c-45f2-8341-aef5e9d6d746, queue=dev-awx-0)
2021-06-06 11:04:04,639 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ef160cdc-dd03-4714-bc2c-907f7559c079, queue=awx_private_queue)
[2021-06-06 11:04:04,639: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ef160cdc-dd03-4714-bc2c-907f7559c079, queue=awx_private_queue)
RESULT 2
OKREADY
RESULT 2
OKREADY
2021-06-06 11:04:05,612 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(af822657-46c9-4a95-9d07-af3e26ea878b, queue=awx_private_queue)
[2021-06-06 11:04:05,612: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(af822657-46c9-4a95-9d07-af3e26ea878b, queue=awx_private_queue)
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
2021-06-06 11:06:31,849 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0e33aff4-1144-4c83-9a98-782c66e3576f, queue=awx_private_queue)
[2021-06-06 11:06:31,849: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0e33aff4-1144-4c83-9a98-782c66e3576f, queue=awx_private_queue)
2021-06-06 11:10:56,702 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a84ac696-9613-4950-85de-61c01880bea1, queue=dev-awx-0)
[2021-06-06 11:10:56,702: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a84ac696-9613-4950-85de-61c01880bea1, queue=dev-awx-0)
2021-06-06 11:10:56,718 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(0a425c1e-87dd-472a-9bbf-498daef2f8a6, queue=dev-awx-0)
[2021-06-06 11:10:56,718: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(0a425c1e-87dd-472a-9bbf-498daef2f8a6, queue=dev-awx-0)
2021-06-06 11:10:56,733 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(86a3a208-52a0-4599-b0ff-0c886cf370ca, queue=awx_private_queue)
[2021-06-06 11:10:56,733: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(86a3a208-52a0-4599-b0ff-0c886cf370ca, queue=awx_private_queue)
2021-06-06 11:10:56,749 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(7a8bad15-d49e-4f7d-8b15-ad4036463672, queue=dev-awx-0)
[2021-06-06 11:10:56,749: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(7a8bad15-d49e-4f7d-8b15-ad4036463672, queue=dev-awx-0)
RESULT 2
OKREADY
2021-06-06 11:11:09,956 DEBUG awx.main.dispatch delivered 234726d5-8096-4154-8eaf-1c0e83332891 to worker[183] qsize 0
2021-06-06 11:11:09,965 DEBUG awx.main.dispatch delivered 6f1b8f30-cf1c-45f2-8341-aef5e9d6d746 to worker[182] qsize 0
2021-06-06 11:11:09,970 DEBUG awx.main.dispatch delivered ef160cdc-dd03-4714-bc2c-907f7559c079 to worker[181] qsize 0
2021-06-06 11:11:09,973 DEBUG awx.main.dispatch delivered af822657-46c9-4a95-9d07-af3e26ea878b to worker[180] qsize 0
2021-06-06 11:11:09,965 DEBUG awx.main.dispatch task 234726d5-8096-4154-8eaf-1c0e83332891 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-06-06 11:11:09,975 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-06-06 11:11:09,969 DEBUG awx.main.dispatch task 6f1b8f30-cf1c-45f2-8341-aef5e9d6d746 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:11:09,975 DEBUG awx.main.dispatch task ef160cdc-dd03-4714-bc2c-907f7559c079 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:11:09,977 DEBUG awx.main.dispatch task af822657-46c9-4a95-9d07-af3e26ea878b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:11:09,982 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:11:09,991 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:11:10,001 WARNING awx.main.dispatch scaling up worker pid:192
2021-06-06 11:11:09,997 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:11:09,999 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_2" does not exist
2021-06-06 11:11:10,008 DEBUG awx.main.dispatch delivered 0e33aff4-1144-4c83-9a98-782c66e3576f to worker[192] qsize 0
2021-06-06 11:11:09,998 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-06-06 11:11:10,002 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:11:10,009 DEBUG awx.main.dispatch task 6f1b8f30-cf1c-45f2-8341-aef5e9d6d746 is finished
2021-06-06 11:11:10,009 DEBUG awx.main.dispatch task ef160cdc-dd03-4714-bc2c-907f7559c079 is finished
2021-06-06 11:11:10,010 DEBUG awx.main.dispatch task 234726d5-8096-4154-8eaf-1c0e83332891 is finished
2021-06-06 11:11:10,010 DEBUG awx.main.dispatch task af822657-46c9-4a95-9d07-af3e26ea878b is finished
2021-06-06 11:11:10,013 WARNING awx.main.dispatch scaling down worker pid:180
2021-06-06 11:11:10,015 WARNING awx.main.dispatch worker exiting gracefully pid:180
2021-06-06 11:11:10,021 DEBUG awx.main.dispatch task 0e33aff4-1144-4c83-9a98-782c66e3576f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:11:10,031 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:11:10,035 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:11:10,033 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 11:11:10,040 DEBUG awx.main.dispatch task 0e33aff4-1144-4c83-9a98-782c66e3576f is finished
2021-06-06 11:11:10,041 DEBUG awx.main.dispatch delivered 0a425c1e-87dd-472a-9bbf-498daef2f8a6 to worker[181] qsize 0
2021-06-06 11:11:10,043 DEBUG awx.main.dispatch delivered 86a3a208-52a0-4599-b0ff-0c886cf370ca to worker[182] qsize 0
2021-06-06 11:11:10,045 DEBUG awx.main.dispatch delivered 7a8bad15-d49e-4f7d-8b15-ad4036463672 to worker[192] qsize 0
2021-06-06 11:11:10,046 DEBUG awx.main.dispatch task 86a3a208-52a0-4599-b0ff-0c886cf370ca starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:11:10,047 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:11:10,047 DEBUG awx.main.dispatch task 7a8bad15-d49e-4f7d-8b15-ad4036463672 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-06-06 11:11:10,050 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-06-06 11:11:10,044 DEBUG awx.main.dispatch task 0a425c1e-87dd-472a-9bbf-498daef2f8a6 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:11:10,055 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:11:10,056 DEBUG awx.main.dispatch task 86a3a208-52a0-4599-b0ff-0c886cf370ca is finished
2021-06-06 11:11:10,058 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_4" does not exist
2021-06-06 11:11:10,060 DEBUG awx.main.dispatch task 0a425c1e-87dd-472a-9bbf-498daef2f8a6 is finished
2021-06-06 11:11:10,066 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 470, in awx_isolated_heartbeat
for isolated_instance in isolated_instance_qs:
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-06-06 11:11:10,068 DEBUG awx.main.dispatch task 7a8bad15-d49e-4f7d-8b15-ad4036463672 is finished
2021-06-06 11:11:16,746 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c318aa2b-5013-4df8-afc2-a6911c3ae54c, queue=awx_private_queue)
[2021-06-06 11:11:16,746: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c318aa2b-5013-4df8-afc2-a6911c3ae54c, queue=awx_private_queue)
2021-06-06 11:11:16,764 DEBUG awx.main.dispatch delivered c318aa2b-5013-4df8-afc2-a6911c3ae54c to worker[182] qsize 0
2021-06-06 11:11:16,765 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bfa50c1d-75ec-456d-a314-60ee248bfa08, queue=awx_private_queue)
[2021-06-06 11:11:16,765: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bfa50c1d-75ec-456d-a314-60ee248bfa08, queue=awx_private_queue)
2021-06-06 11:11:16,768 DEBUG awx.main.dispatch task c318aa2b-5013-4df8-afc2-a6911c3ae54c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:11:16,769 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:11:16,777 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:11:16,780 DEBUG awx.main.dispatch task c318aa2b-5013-4df8-afc2-a6911c3ae54c is finished
2021-06-06 11:11:16,798 DEBUG awx.main.dispatch delivered bfa50c1d-75ec-456d-a314-60ee248bfa08 to worker[192] qsize 0
2021-06-06 11:11:16,800 DEBUG awx.main.dispatch task bfa50c1d-75ec-456d-a314-60ee248bfa08 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:11:16,812 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:11:16,815 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:11:16,817 DEBUG awx.main.dispatch task bfa50c1d-75ec-456d-a314-60ee248bfa08 is finished
2021-06-06 11:11:36,754 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c3559d59-d7ae-4b60-b5cc-28b9fed4323f, queue=awx_private_queue)
[2021-06-06 11:11:36,754: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c3559d59-d7ae-4b60-b5cc-28b9fed4323f, queue=awx_private_queue)
2021-06-06 11:11:36,776 DEBUG awx.main.dispatch delivered c3559d59-d7ae-4b60-b5cc-28b9fed4323f to worker[192] qsize 0
2021-06-06 11:11:36,783 DEBUG awx.main.dispatch task c3559d59-d7ae-4b60-b5cc-28b9fed4323f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:11:36,784 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:11:36,800 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:11:36,802 DEBUG awx.main.dispatch task c3559d59-d7ae-4b60-b5cc-28b9fed4323f is finished
2021-06-06 11:11:46,775 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(cc42de6f-3a42-4b08-88ce-d9405ff0d2f8, queue=awx_private_queue)
[2021-06-06 11:11:46,775: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(cc42de6f-3a42-4b08-88ce-d9405ff0d2f8, queue=awx_private_queue)
2021-06-06 11:11:46,807 DEBUG awx.main.dispatch delivered cc42de6f-3a42-4b08-88ce-d9405ff0d2f8 to worker[183] qsize 0
2021-06-06 11:11:46,823 DEBUG awx.main.dispatch task cc42de6f-3a42-4b08-88ce-d9405ff0d2f8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:11:46,839 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:11:46,847 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:11:46,850 DEBUG awx.main.dispatch task cc42de6f-3a42-4b08-88ce-d9405ff0d2f8 is finished
2021-06-06 11:11:56,702 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a1eb6f44-4271-4a9a-9c19-1d0db27702a1, queue=dev-awx-0)
[2021-06-06 11:11:56,702: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a1eb6f44-4271-4a9a-9c19-1d0db27702a1, queue=dev-awx-0)
2021-06-06 11:11:56,739 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3778309c-1025-4062-b9ae-2bb8ee3a2a2e, queue=dev-awx-0)
[2021-06-06 11:11:56,739: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3778309c-1025-4062-b9ae-2bb8ee3a2a2e, queue=dev-awx-0)
2021-06-06 11:11:56,751 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 11:11:56,779 DEBUG awx.main.dispatch delivered 3778309c-1025-4062-b9ae-2bb8ee3a2a2e to worker[192] qsize 0
2021-06-06 11:11:56,775 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a877f98b-8f07-4037-b860-9e8fda50a2db, queue=awx_private_queue)
[2021-06-06 11:11:56,775: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a877f98b-8f07-4037-b860-9e8fda50a2db, queue=awx_private_queue)
2021-06-06 11:11:56,806 DEBUG awx.main.dispatch delivered a877f98b-8f07-4037-b860-9e8fda50a2db to worker[183] qsize 0
2021-06-06 11:11:56,787 DEBUG awx.main.dispatch task 3778309c-1025-4062-b9ae-2bb8ee3a2a2e starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:11:56,826 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_1" does not exist
2021-06-06 11:11:56,831 DEBUG awx.main.dispatch task 3778309c-1025-4062-b9ae-2bb8ee3a2a2e is finished
2021-06-06 11:11:56,815 DEBUG awx.main.dispatch task a877f98b-8f07-4037-b860-9e8fda50a2db starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:11:56,841 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:11:56,844 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:11:56,850 DEBUG awx.main.dispatch task a877f98b-8f07-4037-b860-9e8fda50a2db is finished
RESULT 2
OKREADY
2021-06-06 11:12:16,785 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6d5c97c1-48e4-4954-b0ea-d0826cc96b28, queue=awx_private_queue)
[2021-06-06 11:12:16,785: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6d5c97c1-48e4-4954-b0ea-d0826cc96b28, queue=awx_private_queue)
2021-06-06 11:12:16,823 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(90525fda-2bb4-4698-86dd-88830fae4f1d, queue=awx_private_queue)
[2021-06-06 11:12:16,823: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(90525fda-2bb4-4698-86dd-88830fae4f1d, queue=awx_private_queue)
2021-06-06 11:12:16,822 DEBUG awx.main.dispatch delivered 6d5c97c1-48e4-4954-b0ea-d0826cc96b28 to worker[181] qsize 0
2021-06-06 11:12:16,867 DEBUG awx.main.dispatch delivered 90525fda-2bb4-4698-86dd-88830fae4f1d to worker[192] qsize 0
2021-06-06 11:12:16,844 DEBUG awx.main.dispatch task 6d5c97c1-48e4-4954-b0ea-d0826cc96b28 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:12:16,878 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:12:16,880 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:12:16,885 DEBUG awx.main.dispatch task 6d5c97c1-48e4-4954-b0ea-d0826cc96b28 is finished
2021-06-06 11:12:16,877 DEBUG awx.main.dispatch task 90525fda-2bb4-4698-86dd-88830fae4f1d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:12:16,893 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:12:16,895 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:12:16,899 DEBUG awx.main.dispatch task 90525fda-2bb4-4698-86dd-88830fae4f1d is finished
2021-06-06 11:12:36,792 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(48f96944-b5b7-436d-92c2-051fdcdfc994, queue=awx_private_queue)
[2021-06-06 11:12:36,792: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(48f96944-b5b7-436d-92c2-051fdcdfc994, queue=awx_private_queue)
2021-06-06 11:12:36,834 DEBUG awx.main.dispatch delivered 48f96944-b5b7-436d-92c2-051fdcdfc994 to worker[192] qsize 0
2021-06-06 11:12:36,844 DEBUG awx.main.dispatch task 48f96944-b5b7-436d-92c2-051fdcdfc994 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:12:36,856 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:12:36,859 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:12:36,863 DEBUG awx.main.dispatch task 48f96944-b5b7-436d-92c2-051fdcdfc994 is finished
2021-06-06 11:12:46,824 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e3277835-f725-487a-a96d-77650ff2e61f, queue=awx_private_queue)
[2021-06-06 11:12:46,824: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e3277835-f725-487a-a96d-77650ff2e61f, queue=awx_private_queue)
2021-06-06 11:12:46,857 DEBUG awx.main.dispatch delivered e3277835-f725-487a-a96d-77650ff2e61f to worker[182] qsize 0
2021-06-06 11:12:46,866 DEBUG awx.main.dispatch task e3277835-f725-487a-a96d-77650ff2e61f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:12:46,893 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:12:46,897 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:12:46,905 DEBUG awx.main.dispatch task e3277835-f725-487a-a96d-77650ff2e61f is finished
2021-06-06 11:12:56,704 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(3a9e3259-3065-4119-b2cf-d2b16b6658dc, queue=dev-awx-0)
[2021-06-06 11:12:56,704: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(3a9e3259-3065-4119-b2cf-d2b16b6658dc, queue=dev-awx-0)
2021-06-06 11:12:56,738 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f1832291-b5ec-4743-b026-7fdc3bab29f6, queue=dev-awx-0)
[2021-06-06 11:12:56,738: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f1832291-b5ec-4743-b026-7fdc3bab29f6, queue=dev-awx-0)
2021-06-06 11:12:56,740 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 11:12:56,768 DEBUG awx.main.dispatch delivered f1832291-b5ec-4743-b026-7fdc3bab29f6 to worker[183] qsize 0
2021-06-06 11:12:56,777 DEBUG awx.main.dispatch task f1832291-b5ec-4743-b026-7fdc3bab29f6 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:12:56,804 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_4" does not exist
2021-06-06 11:12:56,810 DEBUG awx.main.dispatch task f1832291-b5ec-4743-b026-7fdc3bab29f6 is finished
2021-06-06 11:12:56,799 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f091ddbd-5e83-4fa5-ab68-3103b5b497a6, queue=awx_private_queue)
[2021-06-06 11:12:56,799: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f091ddbd-5e83-4fa5-ab68-3103b5b497a6, queue=awx_private_queue)
2021-06-06 11:12:56,832 DEBUG awx.main.dispatch delivered f091ddbd-5e83-4fa5-ab68-3103b5b497a6 to worker[183] qsize 0
2021-06-06 11:12:56,839 DEBUG awx.main.dispatch task f091ddbd-5e83-4fa5-ab68-3103b5b497a6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:12:56,852 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:12:56,854 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:12:56,858 DEBUG awx.main.dispatch task f091ddbd-5e83-4fa5-ab68-3103b5b497a6 is finished
RESULT 2
OKREADY
2021-06-06 11:13:16,805 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(69819e92-fa5f-4249-a533-ba7ae753536d, queue=awx_private_queue)
[2021-06-06 11:13:16,805: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(69819e92-fa5f-4249-a533-ba7ae753536d, queue=awx_private_queue)
2021-06-06 11:13:16,838 DEBUG awx.main.dispatch delivered 69819e92-fa5f-4249-a533-ba7ae753536d to worker[183] qsize 0
2021-06-06 11:13:16,846 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(13161dd8-a600-45db-8720-1c70711d6cb4, queue=awx_private_queue)
[2021-06-06 11:13:16,846: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(13161dd8-a600-45db-8720-1c70711d6cb4, queue=awx_private_queue)
2021-06-06 11:13:16,851 DEBUG awx.main.dispatch task 69819e92-fa5f-4249-a533-ba7ae753536d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:13:16,861 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:13:16,862 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:13:16,867 DEBUG awx.main.dispatch task 69819e92-fa5f-4249-a533-ba7ae753536d is finished
2021-06-06 11:13:16,878 DEBUG awx.main.dispatch delivered 13161dd8-a600-45db-8720-1c70711d6cb4 to worker[182] qsize 0
2021-06-06 11:13:16,905 DEBUG awx.main.dispatch task 13161dd8-a600-45db-8720-1c70711d6cb4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:13:16,922 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:13:16,924 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:13:16,929 DEBUG awx.main.dispatch task 13161dd8-a600-45db-8720-1c70711d6cb4 is finished
2021-06-06 11:13:36,818 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6325b44c-4a8e-49dd-aafe-4b63ea2203b3, queue=awx_private_queue)
[2021-06-06 11:13:36,818: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6325b44c-4a8e-49dd-aafe-4b63ea2203b3, queue=awx_private_queue)
2021-06-06 11:13:36,861 DEBUG awx.main.dispatch delivered 6325b44c-4a8e-49dd-aafe-4b63ea2203b3 to worker[181] qsize 0
2021-06-06 11:13:36,878 DEBUG awx.main.dispatch task 6325b44c-4a8e-49dd-aafe-4b63ea2203b3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:13:36,898 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:13:36,900 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:13:36,906 DEBUG awx.main.dispatch task 6325b44c-4a8e-49dd-aafe-4b63ea2203b3 is finished
2021-06-06 11:13:46,849 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(21542f92-aa72-485a-843c-10dd96aa8cb2, queue=awx_private_queue)
[2021-06-06 11:13:46,849: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(21542f92-aa72-485a-843c-10dd96aa8cb2, queue=awx_private_queue)
2021-06-06 11:13:46,878 DEBUG awx.main.dispatch delivered 21542f92-aa72-485a-843c-10dd96aa8cb2 to worker[192] qsize 0
2021-06-06 11:13:46,890 DEBUG awx.main.dispatch task 21542f92-aa72-485a-843c-10dd96aa8cb2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:13:46,918 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:13:46,921 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:13:46,929 DEBUG awx.main.dispatch task 21542f92-aa72-485a-843c-10dd96aa8cb2 is finished
2021-06-06 11:13:56,704 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(ff89d34a-8b8f-45d3-b1e9-dc705d752dde, queue=dev-awx-0)
[2021-06-06 11:13:56,704: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(ff89d34a-8b8f-45d3-b1e9-dc705d752dde, queue=dev-awx-0)
2021-06-06 11:13:56,746 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b1fe7da5-3b62-4489-bd37-478b6f67068f, queue=dev-awx-0)
[2021-06-06 11:13:56,746: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b1fe7da5-3b62-4489-bd37-478b6f67068f, queue=dev-awx-0)
2021-06-06 11:13:56,755 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 11:13:56,782 DEBUG awx.main.dispatch delivered b1fe7da5-3b62-4489-bd37-478b6f67068f to worker[192] qsize 0
2021-06-06 11:13:56,789 DEBUG awx.main.dispatch task b1fe7da5-3b62-4489-bd37-478b6f67068f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:13:56,804 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_2" does not exist
2021-06-06 11:13:56,811 DEBUG awx.main.dispatch task b1fe7da5-3b62-4489-bd37-478b6f67068f is finished
2021-06-06 11:13:56,814 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b14f7eb3-b4a8-4bb0-81a1-e002558ebea5, queue=awx_private_queue)
[2021-06-06 11:13:56,814: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b14f7eb3-b4a8-4bb0-81a1-e002558ebea5, queue=awx_private_queue)
2021-06-06 11:13:56,847 DEBUG awx.main.dispatch delivered b14f7eb3-b4a8-4bb0-81a1-e002558ebea5 to worker[192] qsize 0
2021-06-06 11:13:56,854 DEBUG awx.main.dispatch task b14f7eb3-b4a8-4bb0-81a1-e002558ebea5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:13:56,865 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:13:56,866 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:13:56,871 DEBUG awx.main.dispatch task b14f7eb3-b4a8-4bb0-81a1-e002558ebea5 is finished
RESULT 2
OKREADY
2021-06-06 11:14:16,827 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2dafb7a3-841e-4e1e-856f-69f282d9d911, queue=awx_private_queue)
[2021-06-06 11:14:16,827: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2dafb7a3-841e-4e1e-856f-69f282d9d911, queue=awx_private_queue)
2021-06-06 11:14:16,852 DEBUG awx.main.dispatch delivered 2dafb7a3-841e-4e1e-856f-69f282d9d911 to worker[192] qsize 0
2021-06-06 11:14:16,855 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8df90c47-ca9f-49a3-b108-a54f092ae22a, queue=awx_private_queue)
[2021-06-06 11:14:16,855: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8df90c47-ca9f-49a3-b108-a54f092ae22a, queue=awx_private_queue)
2021-06-06 11:14:16,861 DEBUG awx.main.dispatch task 2dafb7a3-841e-4e1e-856f-69f282d9d911 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:14:16,872 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:14:16,874 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:14:16,879 DEBUG awx.main.dispatch task 2dafb7a3-841e-4e1e-856f-69f282d9d911 is finished
2021-06-06 11:14:16,890 DEBUG awx.main.dispatch delivered 8df90c47-ca9f-49a3-b108-a54f092ae22a to worker[182] qsize 0
2021-06-06 11:14:16,898 DEBUG awx.main.dispatch task 8df90c47-ca9f-49a3-b108-a54f092ae22a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:14:16,916 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:14:16,919 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:14:16,924 DEBUG awx.main.dispatch task 8df90c47-ca9f-49a3-b108-a54f092ae22a is finished
2021-06-06 11:14:36,837 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(af8c14b4-7ef2-4442-b23a-0edd8a5d775d, queue=awx_private_queue)
[2021-06-06 11:14:36,837: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(af8c14b4-7ef2-4442-b23a-0edd8a5d775d, queue=awx_private_queue)
2021-06-06 11:14:36,864 DEBUG awx.main.dispatch delivered af8c14b4-7ef2-4442-b23a-0edd8a5d775d to worker[182] qsize 0
2021-06-06 11:14:36,870 DEBUG awx.main.dispatch task af8c14b4-7ef2-4442-b23a-0edd8a5d775d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:14:36,884 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:14:36,885 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:14:36,889 DEBUG awx.main.dispatch task af8c14b4-7ef2-4442-b23a-0edd8a5d775d is finished
2021-06-06 11:14:46,863 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f5e9947b-8903-4c56-8f29-a4248ac47c2b, queue=awx_private_queue)
[2021-06-06 11:14:46,863: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f5e9947b-8903-4c56-8f29-a4248ac47c2b, queue=awx_private_queue)
2021-06-06 11:14:46,908 DEBUG awx.main.dispatch delivered f5e9947b-8903-4c56-8f29-a4248ac47c2b to worker[182] qsize 0
2021-06-06 11:14:46,921 DEBUG awx.main.dispatch task f5e9947b-8903-4c56-8f29-a4248ac47c2b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:14:46,937 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:14:46,940 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:14:46,944 DEBUG awx.main.dispatch task f5e9947b-8903-4c56-8f29-a4248ac47c2b is finished
2021-06-06 11:14:56,705 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(94ed2505-6e3f-43c3-8f80-76781563cc80, queue=dev-awx-0)
[2021-06-06 11:14:56,705: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(94ed2505-6e3f-43c3-8f80-76781563cc80, queue=dev-awx-0)
2021-06-06 11:14:56,745 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(cd6dd916-1b7f-456f-8588-1b4cab4cea43, queue=dev-awx-0)
[2021-06-06 11:14:56,745: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(cd6dd916-1b7f-456f-8588-1b4cab4cea43, queue=dev-awx-0)
2021-06-06 11:14:56,755 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 11:14:56,772 DEBUG awx.main.dispatch delivered cd6dd916-1b7f-456f-8588-1b4cab4cea43 to worker[181] qsize 0
2021-06-06 11:14:56,783 DEBUG awx.main.dispatch task cd6dd916-1b7f-456f-8588-1b4cab4cea43 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:14:56,806 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_5" does not exist
2021-06-06 11:14:56,811 DEBUG awx.main.dispatch task cd6dd916-1b7f-456f-8588-1b4cab4cea43 is finished
2021-06-06 11:14:56,836 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(391ee0db-80cb-47ca-b9c3-e70127eed138, queue=awx_private_queue)
[2021-06-06 11:14:56,836: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(391ee0db-80cb-47ca-b9c3-e70127eed138, queue=awx_private_queue)
2021-06-06 11:14:56,863 DEBUG awx.main.dispatch delivered 391ee0db-80cb-47ca-b9c3-e70127eed138 to worker[182] qsize 0
2021-06-06 11:14:56,869 DEBUG awx.main.dispatch task 391ee0db-80cb-47ca-b9c3-e70127eed138 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:14:56,883 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:14:56,894 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:14:56,899 DEBUG awx.main.dispatch task 391ee0db-80cb-47ca-b9c3-e70127eed138 is finished
RESULT 2
OKREADY
2021-06-06 11:15:16,844 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c392d8aa-dc43-4715-8179-fdf922543132, queue=awx_private_queue)
[2021-06-06 11:15:16,844: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c392d8aa-dc43-4715-8179-fdf922543132, queue=awx_private_queue)
2021-06-06 11:15:16,879 DEBUG awx.main.dispatch delivered c392d8aa-dc43-4715-8179-fdf922543132 to worker[182] qsize 0
2021-06-06 11:15:16,887 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(70cecf20-1078-4125-84d8-89933da3eab1, queue=awx_private_queue)
[2021-06-06 11:15:16,887: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(70cecf20-1078-4125-84d8-89933da3eab1, queue=awx_private_queue)
2021-06-06 11:15:16,895 DEBUG awx.main.dispatch task c392d8aa-dc43-4715-8179-fdf922543132 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:15:16,905 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:15:16,906 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:15:16,913 DEBUG awx.main.dispatch task c392d8aa-dc43-4715-8179-fdf922543132 is finished
2021-06-06 11:15:16,921 DEBUG awx.main.dispatch delivered 70cecf20-1078-4125-84d8-89933da3eab1 to worker[182] qsize 0
2021-06-06 11:15:16,929 DEBUG awx.main.dispatch task 70cecf20-1078-4125-84d8-89933da3eab1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:15:16,941 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:15:16,943 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:15:16,947 DEBUG awx.main.dispatch task 70cecf20-1078-4125-84d8-89933da3eab1 is finished
2021-06-06 11:15:36,851 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0fc433a6-a04e-49a6-9595-cbd5e7395f3d, queue=awx_private_queue)
[2021-06-06 11:15:36,851: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0fc433a6-a04e-49a6-9595-cbd5e7395f3d, queue=awx_private_queue)
2021-06-06 11:15:36,877 DEBUG awx.main.dispatch delivered 0fc433a6-a04e-49a6-9595-cbd5e7395f3d to worker[183] qsize 0
2021-06-06 11:15:36,886 DEBUG awx.main.dispatch task 0fc433a6-a04e-49a6-9595-cbd5e7395f3d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:15:36,906 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:15:36,908 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:15:36,913 DEBUG awx.main.dispatch task 0fc433a6-a04e-49a6-9595-cbd5e7395f3d is finished
2021-06-06 11:15:46,879 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ccc66d53-5485-4341-9dcf-770164c13d17, queue=awx_private_queue)
[2021-06-06 11:15:46,879: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ccc66d53-5485-4341-9dcf-770164c13d17, queue=awx_private_queue)
2021-06-06 11:15:46,907 DEBUG awx.main.dispatch delivered ccc66d53-5485-4341-9dcf-770164c13d17 to worker[183] qsize 0
2021-06-06 11:15:46,916 DEBUG awx.main.dispatch task ccc66d53-5485-4341-9dcf-770164c13d17 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:15:46,928 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:15:46,930 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:15:46,935 DEBUG awx.main.dispatch task ccc66d53-5485-4341-9dcf-770164c13d17 is finished
2021-06-06 11:15:56,705 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(650134e8-df1e-4b4e-bf4c-06a613a456b6, queue=dev-awx-0)
[2021-06-06 11:15:56,705: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(650134e8-df1e-4b4e-bf4c-06a613a456b6, queue=dev-awx-0)
2021-06-06 11:15:56,739 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 11:15:56,745 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(316d8c97-0f5e-49d6-8927-8118f990bfc0, queue=dev-awx-0)
[2021-06-06 11:15:56,745: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(316d8c97-0f5e-49d6-8927-8118f990bfc0, queue=dev-awx-0)
2021-06-06 11:15:56,775 DEBUG awx.main.dispatch delivered 316d8c97-0f5e-49d6-8927-8118f990bfc0 to worker[183] qsize 0
2021-06-06 11:15:56,782 DEBUG awx.main.dispatch task 316d8c97-0f5e-49d6-8927-8118f990bfc0 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:15:56,800 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_5" does not exist
2021-06-06 11:15:56,806 DEBUG awx.main.dispatch task 316d8c97-0f5e-49d6-8927-8118f990bfc0 is finished
2021-06-06 11:15:56,851 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4956c0e2-3a94-4159-9088-c7286bd7f182, queue=awx_private_queue)
[2021-06-06 11:15:56,851: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4956c0e2-3a94-4159-9088-c7286bd7f182, queue=awx_private_queue)
2021-06-06 11:15:56,891 DEBUG awx.main.dispatch delivered 4956c0e2-3a94-4159-9088-c7286bd7f182 to worker[183] qsize 0
2021-06-06 11:15:56,898 DEBUG awx.main.dispatch task 4956c0e2-3a94-4159-9088-c7286bd7f182 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:15:56,911 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:15:56,913 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:15:56,918 DEBUG awx.main.dispatch task 4956c0e2-3a94-4159-9088-c7286bd7f182 is finished
RESULT 2
OKREADY
2021-06-06 11:16:16,860 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b826ae6f-6722-41ab-b86b-80c65ccfb995, queue=awx_private_queue)
[2021-06-06 11:16:16,860: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b826ae6f-6722-41ab-b86b-80c65ccfb995, queue=awx_private_queue)
2021-06-06 11:16:16,894 DEBUG awx.main.dispatch delivered b826ae6f-6722-41ab-b86b-80c65ccfb995 to worker[181] qsize 0
2021-06-06 11:16:16,899 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2b28a24a-5e48-4dde-ab2d-f2cd561136d6, queue=awx_private_queue)
[2021-06-06 11:16:16,899: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2b28a24a-5e48-4dde-ab2d-f2cd561136d6, queue=awx_private_queue)
2021-06-06 11:16:16,923 DEBUG awx.main.dispatch delivered 2b28a24a-5e48-4dde-ab2d-f2cd561136d6 to worker[182] qsize 0
2021-06-06 11:16:16,910 DEBUG awx.main.dispatch task b826ae6f-6722-41ab-b86b-80c65ccfb995 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:16:16,928 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:16:16,930 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:16:16,935 DEBUG awx.main.dispatch task b826ae6f-6722-41ab-b86b-80c65ccfb995 is finished
2021-06-06 11:16:16,929 DEBUG awx.main.dispatch task 2b28a24a-5e48-4dde-ab2d-f2cd561136d6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:16:16,946 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:16:16,948 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:16:16,951 DEBUG awx.main.dispatch task 2b28a24a-5e48-4dde-ab2d-f2cd561136d6 is finished
2021-06-06 11:16:36,877 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a1d63095-e2db-4d64-b645-29ffdf1229c5, queue=awx_private_queue)
[2021-06-06 11:16:36,877: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a1d63095-e2db-4d64-b645-29ffdf1229c5, queue=awx_private_queue)
2021-06-06 11:16:36,923 DEBUG awx.main.dispatch delivered a1d63095-e2db-4d64-b645-29ffdf1229c5 to worker[192] qsize 0
2021-06-06 11:16:36,937 DEBUG awx.main.dispatch task a1d63095-e2db-4d64-b645-29ffdf1229c5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:16:36,961 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:16:36,964 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:16:36,971 DEBUG awx.main.dispatch task a1d63095-e2db-4d64-b645-29ffdf1229c5 is finished
2021-06-06 11:16:46,897 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(51c22c87-8f4f-4f8e-877f-8afc5fd23876, queue=awx_private_queue)
[2021-06-06 11:16:46,897: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(51c22c87-8f4f-4f8e-877f-8afc5fd23876, queue=awx_private_queue)
2021-06-06 11:16:46,928 DEBUG awx.main.dispatch delivered 51c22c87-8f4f-4f8e-877f-8afc5fd23876 to worker[183] qsize 0
2021-06-06 11:16:46,941 DEBUG awx.main.dispatch task 51c22c87-8f4f-4f8e-877f-8afc5fd23876 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:16:46,964 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:16:46,967 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:16:46,974 DEBUG awx.main.dispatch task 51c22c87-8f4f-4f8e-877f-8afc5fd23876 is finished
2021-06-06 11:16:56,712 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(383b32ee-2b50-49cc-8811-74584ccf5fcc, queue=dev-awx-0)
[2021-06-06 11:16:56,712: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(383b32ee-2b50-49cc-8811-74584ccf5fcc, queue=dev-awx-0)
2021-06-06 11:16:56,747 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(befc9e4b-c2c0-4851-9ab6-e6178afb8220, queue=dev-awx-0)
[2021-06-06 11:16:56,747: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(befc9e4b-c2c0-4851-9ab6-e6178afb8220, queue=dev-awx-0)
2021-06-06 11:16:56,762 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 11:16:56,789 DEBUG awx.main.dispatch delivered befc9e4b-c2c0-4851-9ab6-e6178afb8220 to worker[181] qsize 0
2021-06-06 11:16:56,799 DEBUG awx.main.dispatch task befc9e4b-c2c0-4851-9ab6-e6178afb8220 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:16:56,816 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_6" does not exist
2021-06-06 11:16:56,821 DEBUG awx.main.dispatch task befc9e4b-c2c0-4851-9ab6-e6178afb8220 is finished
2021-06-06 11:16:56,869 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(08db421e-93e8-4473-8de2-589a51667f8f, queue=awx_private_queue)
[2021-06-06 11:16:56,869: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(08db421e-93e8-4473-8de2-589a51667f8f, queue=awx_private_queue)
2021-06-06 11:16:56,898 DEBUG awx.main.dispatch delivered 08db421e-93e8-4473-8de2-589a51667f8f to worker[183] qsize 0
2021-06-06 11:16:56,907 DEBUG awx.main.dispatch task 08db421e-93e8-4473-8de2-589a51667f8f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:16:56,922 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:16:56,925 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:16:56,929 DEBUG awx.main.dispatch task 08db421e-93e8-4473-8de2-589a51667f8f is finished
RESULT 2
OKREADY
2021-06-06 11:17:16,883 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(13cf9fa3-48ec-4045-a9b6-e68f30b116a1, queue=awx_private_queue)
[2021-06-06 11:17:16,883: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(13cf9fa3-48ec-4045-a9b6-e68f30b116a1, queue=awx_private_queue)
2021-06-06 11:17:16,919 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1789f1b5-8b10-4537-bc39-9c23290cdad3, queue=awx_private_queue)
[2021-06-06 11:17:16,919: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1789f1b5-8b10-4537-bc39-9c23290cdad3, queue=awx_private_queue)
2021-06-06 11:17:16,918 DEBUG awx.main.dispatch delivered 13cf9fa3-48ec-4045-a9b6-e68f30b116a1 to worker[181] qsize 0
2021-06-06 11:17:16,933 DEBUG awx.main.dispatch task 13cf9fa3-48ec-4045-a9b6-e68f30b116a1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:17:16,946 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:17:16,945 DEBUG awx.main.dispatch delivered 1789f1b5-8b10-4537-bc39-9c23290cdad3 to worker[182] qsize 0
2021-06-06 11:17:16,947 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:17:16,952 DEBUG awx.main.dispatch task 13cf9fa3-48ec-4045-a9b6-e68f30b116a1 is finished
2021-06-06 11:17:16,955 DEBUG awx.main.dispatch task 1789f1b5-8b10-4537-bc39-9c23290cdad3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:17:16,973 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:17:16,976 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:17:16,981 DEBUG awx.main.dispatch task 1789f1b5-8b10-4537-bc39-9c23290cdad3 is finished
2021-06-06 11:17:36,892 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e18de525-4a3d-4a43-90ab-98705c46ff50, queue=awx_private_queue)
[2021-06-06 11:17:36,892: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e18de525-4a3d-4a43-90ab-98705c46ff50, queue=awx_private_queue)
2021-06-06 11:17:36,920 DEBUG awx.main.dispatch delivered e18de525-4a3d-4a43-90ab-98705c46ff50 to worker[183] qsize 0
2021-06-06 11:17:36,930 DEBUG awx.main.dispatch task e18de525-4a3d-4a43-90ab-98705c46ff50 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:17:36,944 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:17:36,946 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:17:36,952 DEBUG awx.main.dispatch task e18de525-4a3d-4a43-90ab-98705c46ff50 is finished
2021-06-06 11:17:46,921 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a8681567-a946-4c4a-ab79-9c8c3f445cd4, queue=awx_private_queue)
[2021-06-06 11:17:46,921: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a8681567-a946-4c4a-ab79-9c8c3f445cd4, queue=awx_private_queue)
2021-06-06 11:17:46,947 DEBUG awx.main.dispatch delivered a8681567-a946-4c4a-ab79-9c8c3f445cd4 to worker[181] qsize 0
2021-06-06 11:17:46,953 DEBUG awx.main.dispatch task a8681567-a946-4c4a-ab79-9c8c3f445cd4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:17:46,966 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:17:46,969 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:17:46,974 DEBUG awx.main.dispatch task a8681567-a946-4c4a-ab79-9c8c3f445cd4 is finished
2021-06-06 11:17:56,709 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8c6b09d0-3bc1-41fa-9006-c2c7399501d3, queue=dev-awx-0)
[2021-06-06 11:17:56,709: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8c6b09d0-3bc1-41fa-9006-c2c7399501d3, queue=dev-awx-0)
2021-06-06 11:17:56,746 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(355bca5b-4837-46c4-8493-5ffdbcc7fad7, queue=dev-awx-0)
[2021-06-06 11:17:56,746: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(355bca5b-4837-46c4-8493-5ffdbcc7fad7, queue=dev-awx-0)
2021-06-06 11:17:56,753 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 11:17:56,775 DEBUG awx.main.dispatch delivered 355bca5b-4837-46c4-8493-5ffdbcc7fad7 to worker[181] qsize 0
2021-06-06 11:17:56,783 DEBUG awx.main.dispatch task 355bca5b-4837-46c4-8493-5ffdbcc7fad7 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:17:56,807 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_7" does not exist
2021-06-06 11:17:56,816 DEBUG awx.main.dispatch task 355bca5b-4837-46c4-8493-5ffdbcc7fad7 is finished
2021-06-06 11:17:56,892 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(70f6802a-1311-482c-acd9-46b8c0f735cd, queue=awx_private_queue)
[2021-06-06 11:17:56,892: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(70f6802a-1311-482c-acd9-46b8c0f735cd, queue=awx_private_queue)
2021-06-06 11:17:56,926 DEBUG awx.main.dispatch delivered 70f6802a-1311-482c-acd9-46b8c0f735cd to worker[192] qsize 0
2021-06-06 11:17:56,934 DEBUG awx.main.dispatch task 70f6802a-1311-482c-acd9-46b8c0f735cd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:17:56,957 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:17:56,959 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:17:56,964 DEBUG awx.main.dispatch task 70f6802a-1311-482c-acd9-46b8c0f735cd is finished
RESULT 2
OKREADY
2021-06-06 11:18:16,904 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fa96293c-45c8-48a3-b394-db49f0908c51, queue=awx_private_queue)
[2021-06-06 11:18:16,904: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fa96293c-45c8-48a3-b394-db49f0908c51, queue=awx_private_queue)
2021-06-06 11:18:16,950 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b313c0f0-37fb-499f-9044-e4f97cec765c, queue=awx_private_queue)
[2021-06-06 11:18:16,950: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b313c0f0-37fb-499f-9044-e4f97cec765c, queue=awx_private_queue)
2021-06-06 11:18:16,950 DEBUG awx.main.dispatch delivered fa96293c-45c8-48a3-b394-db49f0908c51 to worker[182] qsize 0
2021-06-06 11:18:16,968 DEBUG awx.main.dispatch task fa96293c-45c8-48a3-b394-db49f0908c51 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:18:16,987 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:18:16,984 DEBUG awx.main.dispatch delivered b313c0f0-37fb-499f-9044-e4f97cec765c to worker[181] qsize 0
2021-06-06 11:18:16,989 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:18:16,994 DEBUG awx.main.dispatch task fa96293c-45c8-48a3-b394-db49f0908c51 is finished
2021-06-06 11:18:16,993 DEBUG awx.main.dispatch task b313c0f0-37fb-499f-9044-e4f97cec765c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:18:17,017 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:18:17,020 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:18:17,027 DEBUG awx.main.dispatch task b313c0f0-37fb-499f-9044-e4f97cec765c is finished
2021-06-06 11:18:36,911 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(41b4ddef-18e5-4ca3-aae6-5ee06660f5c2, queue=awx_private_queue)
[2021-06-06 11:18:36,911: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(41b4ddef-18e5-4ca3-aae6-5ee06660f5c2, queue=awx_private_queue)
2021-06-06 11:18:36,942 DEBUG awx.main.dispatch delivered 41b4ddef-18e5-4ca3-aae6-5ee06660f5c2 to worker[192] qsize 0
2021-06-06 11:18:36,953 DEBUG awx.main.dispatch task 41b4ddef-18e5-4ca3-aae6-5ee06660f5c2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:18:36,970 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:18:36,973 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:18:36,980 DEBUG awx.main.dispatch task 41b4ddef-18e5-4ca3-aae6-5ee06660f5c2 is finished
2021-06-06 11:18:46,951 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4bb181c0-761d-4c54-9938-d9f460585a07, queue=awx_private_queue)
[2021-06-06 11:18:46,951: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4bb181c0-761d-4c54-9938-d9f460585a07, queue=awx_private_queue)
2021-06-06 11:18:46,978 DEBUG awx.main.dispatch delivered 4bb181c0-761d-4c54-9938-d9f460585a07 to worker[181] qsize 0
2021-06-06 11:18:46,986 DEBUG awx.main.dispatch task 4bb181c0-761d-4c54-9938-d9f460585a07 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:18:47,005 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:18:47,010 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:18:47,017 DEBUG awx.main.dispatch task 4bb181c0-761d-4c54-9938-d9f460585a07 is finished
2021-06-06 11:18:56,704 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(656eb35f-a0d3-4262-999b-df386336f17e, queue=dev-awx-0)
[2021-06-06 11:18:56,704: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(656eb35f-a0d3-4262-999b-df386336f17e, queue=dev-awx-0)
2021-06-06 11:18:56,734 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 11:18:56,760 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c6472a50-cb8b-4f38-89cd-9a290cf9bff2, queue=dev-awx-0)
[2021-06-06 11:18:56,760: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c6472a50-cb8b-4f38-89cd-9a290cf9bff2, queue=dev-awx-0)
2021-06-06 11:18:56,796 DEBUG awx.main.dispatch delivered c6472a50-cb8b-4f38-89cd-9a290cf9bff2 to worker[182] qsize 0
2021-06-06 11:18:56,807 DEBUG awx.main.dispatch task c6472a50-cb8b-4f38-89cd-9a290cf9bff2 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:18:56,824 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_3" does not exist
2021-06-06 11:18:56,828 DEBUG awx.main.dispatch task c6472a50-cb8b-4f38-89cd-9a290cf9bff2 is finished
2021-06-06 11:18:56,910 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7ce73732-78bc-404c-9c2d-d2e34b86e695, queue=awx_private_queue)
[2021-06-06 11:18:56,910: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7ce73732-78bc-404c-9c2d-d2e34b86e695, queue=awx_private_queue)
2021-06-06 11:18:56,937 DEBUG awx.main.dispatch delivered 7ce73732-78bc-404c-9c2d-d2e34b86e695 to worker[192] qsize 0
2021-06-06 11:18:56,944 DEBUG awx.main.dispatch task 7ce73732-78bc-404c-9c2d-d2e34b86e695 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:18:56,958 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:18:56,959 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:18:56,963 DEBUG awx.main.dispatch task 7ce73732-78bc-404c-9c2d-d2e34b86e695 is finished
RESULT 2
OKREADY
2021-06-06 11:19:16,919 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e7fb3477-8a25-4024-84e7-424803a8af2d, queue=awx_private_queue)
[2021-06-06 11:19:16,919: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e7fb3477-8a25-4024-84e7-424803a8af2d, queue=awx_private_queue)
2021-06-06 11:19:16,945 DEBUG awx.main.dispatch delivered e7fb3477-8a25-4024-84e7-424803a8af2d to worker[183] qsize 0
2021-06-06 11:19:16,951 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9b1a6a13-0313-4039-9596-85d5b59bde1d, queue=awx_private_queue)
[2021-06-06 11:19:16,951: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9b1a6a13-0313-4039-9596-85d5b59bde1d, queue=awx_private_queue)
2021-06-06 11:19:16,955 DEBUG awx.main.dispatch task e7fb3477-8a25-4024-84e7-424803a8af2d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:19:16,979 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:19:16,980 DEBUG awx.main.dispatch delivered 9b1a6a13-0313-4039-9596-85d5b59bde1d to worker[182] qsize 0
2021-06-06 11:19:16,981 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:19:16,991 DEBUG awx.main.dispatch task e7fb3477-8a25-4024-84e7-424803a8af2d is finished
2021-06-06 11:19:16,989 DEBUG awx.main.dispatch task 9b1a6a13-0313-4039-9596-85d5b59bde1d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:19:17,007 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:19:17,010 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:19:17,013 DEBUG awx.main.dispatch task 9b1a6a13-0313-4039-9596-85d5b59bde1d is finished
2021-06-06 11:19:36,937 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(da7c967f-b6bb-4d07-92e8-a0e72b7d90bc, queue=awx_private_queue)
[2021-06-06 11:19:36,937: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(da7c967f-b6bb-4d07-92e8-a0e72b7d90bc, queue=awx_private_queue)
2021-06-06 11:19:36,995 DEBUG awx.main.dispatch delivered da7c967f-b6bb-4d07-92e8-a0e72b7d90bc to worker[181] qsize 0
2021-06-06 11:19:37,012 DEBUG awx.main.dispatch task da7c967f-b6bb-4d07-92e8-a0e72b7d90bc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:19:37,032 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:19:37,034 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:19:37,041 DEBUG awx.main.dispatch task da7c967f-b6bb-4d07-92e8-a0e72b7d90bc is finished
2021-06-06 11:19:46,952 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2b67a88f-d07e-495d-964b-05ce9fa0b6f8, queue=awx_private_queue)
[2021-06-06 11:19:46,952: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2b67a88f-d07e-495d-964b-05ce9fa0b6f8, queue=awx_private_queue)
2021-06-06 11:19:46,982 DEBUG awx.main.dispatch delivered 2b67a88f-d07e-495d-964b-05ce9fa0b6f8 to worker[181] qsize 0
2021-06-06 11:19:46,992 DEBUG awx.main.dispatch task 2b67a88f-d07e-495d-964b-05ce9fa0b6f8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:19:47,007 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:19:47,010 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:19:47,016 DEBUG awx.main.dispatch task 2b67a88f-d07e-495d-964b-05ce9fa0b6f8 is finished
2021-06-06 11:19:56,706 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(91bf5a1e-dd78-4dbe-9fd8-545001c4380f, queue=dev-awx-0)
[2021-06-06 11:19:56,706: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(91bf5a1e-dd78-4dbe-9fd8-545001c4380f, queue=dev-awx-0)
2021-06-06 11:19:56,742 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 11:19:56,747 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8efb0ece-789d-4515-bc67-006006f814bb, queue=dev-awx-0)
[2021-06-06 11:19:56,747: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8efb0ece-789d-4515-bc67-006006f814bb, queue=dev-awx-0)
2021-06-06 11:19:56,772 DEBUG awx.main.dispatch delivered 8efb0ece-789d-4515-bc67-006006f814bb to worker[183] qsize 0
2021-06-06 11:19:56,780 DEBUG awx.main.dispatch task 8efb0ece-789d-4515-bc67-006006f814bb starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:19:56,804 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_6" does not exist
2021-06-06 11:19:56,809 DEBUG awx.main.dispatch task 8efb0ece-789d-4515-bc67-006006f814bb is finished
2021-06-06 11:19:56,929 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0a19136b-9437-49c8-a024-51f7042c8212, queue=awx_private_queue)
[2021-06-06 11:19:56,929: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0a19136b-9437-49c8-a024-51f7042c8212, queue=awx_private_queue)
2021-06-06 11:19:56,955 DEBUG awx.main.dispatch delivered 0a19136b-9437-49c8-a024-51f7042c8212 to worker[192] qsize 0
2021-06-06 11:19:56,963 DEBUG awx.main.dispatch task 0a19136b-9437-49c8-a024-51f7042c8212 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:19:56,983 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:19:56,985 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:19:56,990 DEBUG awx.main.dispatch task 0a19136b-9437-49c8-a024-51f7042c8212 is finished
RESULT 2
OKREADY
2021-06-06 11:20:16,942 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cbd0a1c6-4047-414f-b2c4-e49bd4a07f25, queue=awx_private_queue)
[2021-06-06 11:20:16,942: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cbd0a1c6-4047-414f-b2c4-e49bd4a07f25, queue=awx_private_queue)
2021-06-06 11:20:16,975 DEBUG awx.main.dispatch delivered cbd0a1c6-4047-414f-b2c4-e49bd4a07f25 to worker[183] qsize 0
2021-06-06 11:20:16,976 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(69cdae58-af34-46f5-b822-16a72fc6096d, queue=awx_private_queue)
[2021-06-06 11:20:16,976: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(69cdae58-af34-46f5-b822-16a72fc6096d, queue=awx_private_queue)
2021-06-06 11:20:16,986 DEBUG awx.main.dispatch task cbd0a1c6-4047-414f-b2c4-e49bd4a07f25 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:20:17,003 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:20:17,005 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:20:17,003 DEBUG awx.main.dispatch delivered 69cdae58-af34-46f5-b822-16a72fc6096d to worker[181] qsize 0
2021-06-06 11:20:17,011 DEBUG awx.main.dispatch task cbd0a1c6-4047-414f-b2c4-e49bd4a07f25 is finished
2021-06-06 11:20:17,014 DEBUG awx.main.dispatch task 69cdae58-af34-46f5-b822-16a72fc6096d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:20:17,030 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:20:17,033 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:20:17,037 DEBUG awx.main.dispatch task 69cdae58-af34-46f5-b822-16a72fc6096d is finished
2021-06-06 11:20:36,952 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(333d2db0-baa8-4716-930f-001eddd6363e, queue=awx_private_queue)
[2021-06-06 11:20:36,952: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(333d2db0-baa8-4716-930f-001eddd6363e, queue=awx_private_queue)
2021-06-06 11:20:36,995 DEBUG awx.main.dispatch delivered 333d2db0-baa8-4716-930f-001eddd6363e to worker[181] qsize 0
2021-06-06 11:20:37,002 DEBUG awx.main.dispatch task 333d2db0-baa8-4716-930f-001eddd6363e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:20:37,020 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:20:37,021 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:20:37,036 DEBUG awx.main.dispatch task 333d2db0-baa8-4716-930f-001eddd6363e is finished
2021-06-06 11:20:46,978 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(49c12e02-ec65-4d98-a891-85777689cfdd, queue=awx_private_queue)
[2021-06-06 11:20:46,978: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(49c12e02-ec65-4d98-a891-85777689cfdd, queue=awx_private_queue)
2021-06-06 11:20:47,015 DEBUG awx.main.dispatch delivered 49c12e02-ec65-4d98-a891-85777689cfdd to worker[183] qsize 0
2021-06-06 11:20:47,022 DEBUG awx.main.dispatch task 49c12e02-ec65-4d98-a891-85777689cfdd starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:20:47,035 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:20:47,037 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:20:47,042 DEBUG awx.main.dispatch task 49c12e02-ec65-4d98-a891-85777689cfdd is finished
2021-06-06 11:20:56,706 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(474ed8c6-73bf-4344-ae8f-2cc23800353c, queue=dev-awx-0)
[2021-06-06 11:20:56,706: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(474ed8c6-73bf-4344-ae8f-2cc23800353c, queue=dev-awx-0)
2021-06-06 11:20:56,742 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 11:20:56,749 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(e71e2bb9-04da-475b-addf-8e5374b30512, queue=dev-awx-0)
[2021-06-06 11:20:56,749: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(e71e2bb9-04da-475b-addf-8e5374b30512, queue=dev-awx-0)
2021-06-06 11:20:56,785 DEBUG awx.main.dispatch delivered e71e2bb9-04da-475b-addf-8e5374b30512 to worker[192] qsize 0
2021-06-06 11:20:56,786 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ac3ae7ec-f60a-425a-9b64-62089c2bd5dd, queue=dev-awx-0)
[2021-06-06 11:20:56,786: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ac3ae7ec-f60a-425a-9b64-62089c2bd5dd, queue=dev-awx-0)
2021-06-06 11:20:56,804 DEBUG awx.main.dispatch task e71e2bb9-04da-475b-addf-8e5374b30512 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-06-06 11:20:56,822 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-06-06 11:20:56,828 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 470, in awx_isolated_heartbeat
for isolated_instance in isolated_instance_qs:
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-06-06 11:20:56,836 DEBUG awx.main.dispatch delivered ac3ae7ec-f60a-425a-9b64-62089c2bd5dd to worker[181] qsize 0
2021-06-06 11:20:56,835 DEBUG awx.main.dispatch task e71e2bb9-04da-475b-addf-8e5374b30512 is finished
2021-06-06 11:20:56,843 DEBUG awx.main.dispatch task ac3ae7ec-f60a-425a-9b64-62089c2bd5dd starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:20:56,855 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_8" does not exist
2021-06-06 11:20:56,859 DEBUG awx.main.dispatch task ac3ae7ec-f60a-425a-9b64-62089c2bd5dd is finished
2021-06-06 11:20:56,951 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(73c7e746-fd66-4932-9367-91ac0b45512d, queue=awx_private_queue)
[2021-06-06 11:20:56,951: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(73c7e746-fd66-4932-9367-91ac0b45512d, queue=awx_private_queue)
2021-06-06 11:20:56,978 DEBUG awx.main.dispatch delivered 73c7e746-fd66-4932-9367-91ac0b45512d to worker[192] qsize 0
2021-06-06 11:20:56,984 DEBUG awx.main.dispatch task 73c7e746-fd66-4932-9367-91ac0b45512d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:20:56,998 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:20:56,999 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:20:57,006 DEBUG awx.main.dispatch task 73c7e746-fd66-4932-9367-91ac0b45512d is finished
RESULT 2
OKREADY
2021-06-06 11:21:16,965 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d0021b1b-8f1e-48eb-9530-21bf49cce4b7, queue=awx_private_queue)
[2021-06-06 11:21:16,965: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d0021b1b-8f1e-48eb-9530-21bf49cce4b7, queue=awx_private_queue)
2021-06-06 11:21:17,000 DEBUG awx.main.dispatch delivered d0021b1b-8f1e-48eb-9530-21bf49cce4b7 to worker[183] qsize 0
2021-06-06 11:21:17,002 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2f63f7fb-6b39-490a-8082-68cce5429793, queue=awx_private_queue)
[2021-06-06 11:21:17,002: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2f63f7fb-6b39-490a-8082-68cce5429793, queue=awx_private_queue)
2021-06-06 11:21:17,025 DEBUG awx.main.dispatch delivered 2f63f7fb-6b39-490a-8082-68cce5429793 to worker[181] qsize 0
2021-06-06 11:21:17,015 DEBUG awx.main.dispatch task d0021b1b-8f1e-48eb-9530-21bf49cce4b7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:21:17,038 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:21:17,040 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:21:17,047 DEBUG awx.main.dispatch task d0021b1b-8f1e-48eb-9530-21bf49cce4b7 is finished
2021-06-06 11:21:17,035 DEBUG awx.main.dispatch task 2f63f7fb-6b39-490a-8082-68cce5429793 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:21:17,058 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:21:17,060 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:21:17,065 DEBUG awx.main.dispatch task 2f63f7fb-6b39-490a-8082-68cce5429793 is finished
2021-06-06 11:21:36,970 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e4f208bd-1d65-495a-8a49-29234990b023, queue=awx_private_queue)
[2021-06-06 11:21:36,970: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e4f208bd-1d65-495a-8a49-29234990b023, queue=awx_private_queue)
2021-06-06 11:21:36,998 DEBUG awx.main.dispatch delivered e4f208bd-1d65-495a-8a49-29234990b023 to worker[183] qsize 0
2021-06-06 11:21:37,009 DEBUG awx.main.dispatch task e4f208bd-1d65-495a-8a49-29234990b023 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:21:37,019 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:21:37,021 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:21:37,025 DEBUG awx.main.dispatch task e4f208bd-1d65-495a-8a49-29234990b023 is finished
2021-06-06 11:21:47,002 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4ee1b8c5-3903-49c1-a971-3d5bd014f440, queue=awx_private_queue)
[2021-06-06 11:21:47,002: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4ee1b8c5-3903-49c1-a971-3d5bd014f440, queue=awx_private_queue)
2021-06-06 11:21:47,027 DEBUG awx.main.dispatch delivered 4ee1b8c5-3903-49c1-a971-3d5bd014f440 to worker[183] qsize 0
2021-06-06 11:21:47,035 DEBUG awx.main.dispatch task 4ee1b8c5-3903-49c1-a971-3d5bd014f440 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:21:47,055 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:21:47,058 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:21:47,065 DEBUG awx.main.dispatch task 4ee1b8c5-3903-49c1-a971-3d5bd014f440 is finished
2021-06-06 11:21:56,707 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5bccf5c3-5bdd-41df-9f1f-c840a18ac2da, queue=dev-awx-0)
[2021-06-06 11:21:56,707: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5bccf5c3-5bdd-41df-9f1f-c840a18ac2da, queue=dev-awx-0)
2021-06-06 11:21:56,753 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 11:21:56,797 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(43aa01ba-85aa-4cc4-87a8-c14f587e175e, queue=dev-awx-0)
[2021-06-06 11:21:56,797: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(43aa01ba-85aa-4cc4-87a8-c14f587e175e, queue=dev-awx-0)
2021-06-06 11:21:56,833 DEBUG awx.main.dispatch delivered 43aa01ba-85aa-4cc4-87a8-c14f587e175e to worker[183] qsize 0
2021-06-06 11:21:56,839 DEBUG awx.main.dispatch task 43aa01ba-85aa-4cc4-87a8-c14f587e175e starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:21:56,858 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_7" does not exist
2021-06-06 11:21:56,866 DEBUG awx.main.dispatch task 43aa01ba-85aa-4cc4-87a8-c14f587e175e is finished
2021-06-06 11:21:56,970 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4419dc1d-7f5b-42e2-afbd-53f42cb6a8b0, queue=awx_private_queue)
[2021-06-06 11:21:56,970: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4419dc1d-7f5b-42e2-afbd-53f42cb6a8b0, queue=awx_private_queue)
2021-06-06 11:21:57,001 DEBUG awx.main.dispatch delivered 4419dc1d-7f5b-42e2-afbd-53f42cb6a8b0 to worker[192] qsize 0
2021-06-06 11:21:57,012 DEBUG awx.main.dispatch task 4419dc1d-7f5b-42e2-afbd-53f42cb6a8b0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:21:57,035 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:21:57,037 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:21:57,041 DEBUG awx.main.dispatch task 4419dc1d-7f5b-42e2-afbd-53f42cb6a8b0 is finished
RESULT 2
OKREADY
2021-06-06 11:22:16,979 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cec75e51-d0a3-415b-84d2-b830eac4c68f, queue=awx_private_queue)
[2021-06-06 11:22:16,979: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cec75e51-d0a3-415b-84d2-b830eac4c68f, queue=awx_private_queue)
2021-06-06 11:22:17,033 DEBUG awx.main.dispatch delivered cec75e51-d0a3-415b-84d2-b830eac4c68f to worker[192] qsize 0
2021-06-06 11:22:17,035 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8c1d55ba-64ad-4257-9d2f-ef84e44e7829, queue=awx_private_queue)
[2021-06-06 11:22:17,035: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8c1d55ba-64ad-4257-9d2f-ef84e44e7829, queue=awx_private_queue)
2021-06-06 11:22:17,044 DEBUG awx.main.dispatch task cec75e51-d0a3-415b-84d2-b830eac4c68f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:22:17,058 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:22:17,062 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:22:17,065 DEBUG awx.main.dispatch delivered 8c1d55ba-64ad-4257-9d2f-ef84e44e7829 to worker[182] qsize 0
2021-06-06 11:22:17,069 DEBUG awx.main.dispatch task cec75e51-d0a3-415b-84d2-b830eac4c68f is finished
2021-06-06 11:22:17,077 DEBUG awx.main.dispatch task 8c1d55ba-64ad-4257-9d2f-ef84e44e7829 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:22:17,103 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:22:17,105 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:22:17,110 DEBUG awx.main.dispatch task 8c1d55ba-64ad-4257-9d2f-ef84e44e7829 is finished
2021-06-06 11:22:36,983 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a5032937-3f68-4f25-b8ce-9be5d7304c91, queue=awx_private_queue)
[2021-06-06 11:22:36,983: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a5032937-3f68-4f25-b8ce-9be5d7304c91, queue=awx_private_queue)
2021-06-06 11:22:37,011 DEBUG awx.main.dispatch delivered a5032937-3f68-4f25-b8ce-9be5d7304c91 to worker[183] qsize 0
2021-06-06 11:22:37,017 DEBUG awx.main.dispatch task a5032937-3f68-4f25-b8ce-9be5d7304c91 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:22:37,031 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:22:37,033 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:22:37,038 DEBUG awx.main.dispatch task a5032937-3f68-4f25-b8ce-9be5d7304c91 is finished
2021-06-06 11:22:47,043 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(79e092c3-6255-4e88-9d3e-e7315eb92786, queue=awx_private_queue)
[2021-06-06 11:22:47,043: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(79e092c3-6255-4e88-9d3e-e7315eb92786, queue=awx_private_queue)
2021-06-06 11:22:47,083 DEBUG awx.main.dispatch delivered 79e092c3-6255-4e88-9d3e-e7315eb92786 to worker[182] qsize 0
2021-06-06 11:22:47,094 DEBUG awx.main.dispatch task 79e092c3-6255-4e88-9d3e-e7315eb92786 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:22:47,123 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:22:47,126 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:22:47,131 DEBUG awx.main.dispatch task 79e092c3-6255-4e88-9d3e-e7315eb92786 is finished
2021-06-06 11:22:56,707 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d1062158-64bd-4044-82d3-098792a868a5, queue=dev-awx-0)
[2021-06-06 11:22:56,707: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d1062158-64bd-4044-82d3-098792a868a5, queue=dev-awx-0)
2021-06-06 11:22:56,742 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 11:22:56,787 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(96e448a7-4e33-4586-92cf-5d7b02a4acdb, queue=dev-awx-0)
[2021-06-06 11:22:56,787: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(96e448a7-4e33-4586-92cf-5d7b02a4acdb, queue=dev-awx-0)
2021-06-06 11:22:56,812 DEBUG awx.main.dispatch delivered 96e448a7-4e33-4586-92cf-5d7b02a4acdb to worker[182] qsize 0
2021-06-06 11:22:56,823 DEBUG awx.main.dispatch task 96e448a7-4e33-4586-92cf-5d7b02a4acdb starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:22:56,835 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_4" does not exist
2021-06-06 11:22:56,839 DEBUG awx.main.dispatch task 96e448a7-4e33-4586-92cf-5d7b02a4acdb is finished
2021-06-06 11:22:56,983 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a3c79475-ef8d-4211-8989-38c47f73a7f8, queue=awx_private_queue)
[2021-06-06 11:22:56,983: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a3c79475-ef8d-4211-8989-38c47f73a7f8, queue=awx_private_queue)
2021-06-06 11:22:57,008 DEBUG awx.main.dispatch delivered a3c79475-ef8d-4211-8989-38c47f73a7f8 to worker[183] qsize 0
2021-06-06 11:22:57,018 DEBUG awx.main.dispatch task a3c79475-ef8d-4211-8989-38c47f73a7f8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:22:57,033 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:22:57,035 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:22:57,040 DEBUG awx.main.dispatch task a3c79475-ef8d-4211-8989-38c47f73a7f8 is finished
RESULT 2
OKREADY
2021-06-06 11:23:16,995 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c9fe3e6f-b3d5-4bd0-aab8-282a19892958, queue=awx_private_queue)
[2021-06-06 11:23:16,995: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c9fe3e6f-b3d5-4bd0-aab8-282a19892958, queue=awx_private_queue)
2021-06-06 11:23:17,027 DEBUG awx.main.dispatch delivered c9fe3e6f-b3d5-4bd0-aab8-282a19892958 to worker[182] qsize 0
2021-06-06 11:23:17,037 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(55c909d3-a335-4540-92c9-fff9e9c5d5e5, queue=awx_private_queue)
[2021-06-06 11:23:17,037: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(55c909d3-a335-4540-92c9-fff9e9c5d5e5, queue=awx_private_queue)
2021-06-06 11:23:17,039 DEBUG awx.main.dispatch task c9fe3e6f-b3d5-4bd0-aab8-282a19892958 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:23:17,054 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:23:17,059 DEBUG awx.main.dispatch delivered 55c909d3-a335-4540-92c9-fff9e9c5d5e5 to worker[183] qsize 0
2021-06-06 11:23:17,056 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:23:17,069 DEBUG awx.main.dispatch task c9fe3e6f-b3d5-4bd0-aab8-282a19892958 is finished
2021-06-06 11:23:17,068 DEBUG awx.main.dispatch task 55c909d3-a335-4540-92c9-fff9e9c5d5e5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:23:17,081 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:23:17,083 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:23:17,087 DEBUG awx.main.dispatch task 55c909d3-a335-4540-92c9-fff9e9c5d5e5 is finished
2021-06-06 11:23:36,995 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bb68e75a-edb0-48d3-a528-8771f4625650, queue=awx_private_queue)
[2021-06-06 11:23:36,995: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bb68e75a-edb0-48d3-a528-8771f4625650, queue=awx_private_queue)
2021-06-06 11:23:37,025 DEBUG awx.main.dispatch delivered bb68e75a-edb0-48d3-a528-8771f4625650 to worker[183] qsize 0
2021-06-06 11:23:37,033 DEBUG awx.main.dispatch task bb68e75a-edb0-48d3-a528-8771f4625650 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:23:37,047 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:23:37,048 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:23:37,052 DEBUG awx.main.dispatch task bb68e75a-edb0-48d3-a528-8771f4625650 is finished
2021-06-06 11:23:47,037 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f85574c3-938b-4400-9653-d3fc65467857, queue=awx_private_queue)
[2021-06-06 11:23:47,037: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f85574c3-938b-4400-9653-d3fc65467857, queue=awx_private_queue)
2021-06-06 11:23:47,066 DEBUG awx.main.dispatch delivered f85574c3-938b-4400-9653-d3fc65467857 to worker[192] qsize 0
2021-06-06 11:23:47,076 DEBUG awx.main.dispatch task f85574c3-938b-4400-9653-d3fc65467857 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:23:47,097 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:23:47,099 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:23:47,114 DEBUG awx.main.dispatch task f85574c3-938b-4400-9653-d3fc65467857 is finished
2021-06-06 11:23:56,707 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e09c30ba-4ce4-4b22-93f4-1b8f0dd8e672, queue=dev-awx-0)
[2021-06-06 11:23:56,707: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e09c30ba-4ce4-4b22-93f4-1b8f0dd8e672, queue=dev-awx-0)
2021-06-06 11:23:56,742 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 11:23:56,787 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(366ed311-09ee-489e-887f-022ac1a54794, queue=dev-awx-0)
[2021-06-06 11:23:56,787: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(366ed311-09ee-489e-887f-022ac1a54794, queue=dev-awx-0)
2021-06-06 11:23:56,811 DEBUG awx.main.dispatch delivered 366ed311-09ee-489e-887f-022ac1a54794 to worker[182] qsize 0
2021-06-06 11:23:56,828 DEBUG awx.main.dispatch task 366ed311-09ee-489e-887f-022ac1a54794 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:23:56,848 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_5" does not exist
2021-06-06 11:23:56,854 DEBUG awx.main.dispatch task 366ed311-09ee-489e-887f-022ac1a54794 is finished
2021-06-06 11:23:56,995 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(40c633cd-5932-4390-a70c-32fca525a778, queue=awx_private_queue)
[2021-06-06 11:23:56,995: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(40c633cd-5932-4390-a70c-32fca525a778, queue=awx_private_queue)
2021-06-06 11:23:57,023 DEBUG awx.main.dispatch delivered 40c633cd-5932-4390-a70c-32fca525a778 to worker[183] qsize 0
2021-06-06 11:23:57,034 DEBUG awx.main.dispatch task 40c633cd-5932-4390-a70c-32fca525a778 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:23:57,046 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:23:57,048 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:23:57,056 DEBUG awx.main.dispatch task 40c633cd-5932-4390-a70c-32fca525a778 is finished
RESULT 2
OKREADY
2021-06-06 11:24:17,004 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(dba43048-3022-408e-b041-728199bd54ab, queue=awx_private_queue)
[2021-06-06 11:24:17,004: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(dba43048-3022-408e-b041-728199bd54ab, queue=awx_private_queue)
2021-06-06 11:24:17,040 DEBUG awx.main.dispatch delivered dba43048-3022-408e-b041-728199bd54ab to worker[181] qsize 0
2021-06-06 11:24:17,042 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(57c39fae-89a9-4c96-8a7a-f4b82d1c2b23, queue=awx_private_queue)
[2021-06-06 11:24:17,042: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(57c39fae-89a9-4c96-8a7a-f4b82d1c2b23, queue=awx_private_queue)
2021-06-06 11:24:17,049 DEBUG awx.main.dispatch task dba43048-3022-408e-b041-728199bd54ab starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:24:17,067 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:24:17,064 DEBUG awx.main.dispatch delivered 57c39fae-89a9-4c96-8a7a-f4b82d1c2b23 to worker[183] qsize 0
2021-06-06 11:24:17,069 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:24:17,074 DEBUG awx.main.dispatch task dba43048-3022-408e-b041-728199bd54ab is finished
2021-06-06 11:24:17,072 DEBUG awx.main.dispatch task 57c39fae-89a9-4c96-8a7a-f4b82d1c2b23 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:24:17,084 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:24:17,087 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:24:17,093 DEBUG awx.main.dispatch task 57c39fae-89a9-4c96-8a7a-f4b82d1c2b23 is finished
2021-06-06 11:24:37,020 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7bc521fc-8576-496b-8e32-2aedfc6dc840, queue=awx_private_queue)
[2021-06-06 11:24:37,020: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7bc521fc-8576-496b-8e32-2aedfc6dc840, queue=awx_private_queue)
2021-06-06 11:24:37,055 DEBUG awx.main.dispatch delivered 7bc521fc-8576-496b-8e32-2aedfc6dc840 to worker[192] qsize 0
2021-06-06 11:24:37,067 DEBUG awx.main.dispatch task 7bc521fc-8576-496b-8e32-2aedfc6dc840 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:24:37,084 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:24:37,085 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:24:37,090 DEBUG awx.main.dispatch task 7bc521fc-8576-496b-8e32-2aedfc6dc840 is finished
2021-06-06 11:24:47,044 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9aee48a2-e0a5-4980-832e-ad091a0d4d39, queue=awx_private_queue)
[2021-06-06 11:24:47,044: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9aee48a2-e0a5-4980-832e-ad091a0d4d39, queue=awx_private_queue)
2021-06-06 11:24:47,078 DEBUG awx.main.dispatch delivered 9aee48a2-e0a5-4980-832e-ad091a0d4d39 to worker[182] qsize 0
2021-06-06 11:24:47,089 DEBUG awx.main.dispatch task 9aee48a2-e0a5-4980-832e-ad091a0d4d39 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:24:47,116 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:24:47,119 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:24:47,125 DEBUG awx.main.dispatch task 9aee48a2-e0a5-4980-832e-ad091a0d4d39 is finished
2021-06-06 11:24:56,708 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(ca2e25ec-5120-495d-ab7c-16eaacefc630, queue=dev-awx-0)
[2021-06-06 11:24:56,708: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(ca2e25ec-5120-495d-ab7c-16eaacefc630, queue=dev-awx-0)
2021-06-06 11:24:56,743 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-06-06 11:24:56,786 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8da66ddf-df14-4a27-ae54-afe9dc2d15af, queue=dev-awx-0)
[2021-06-06 11:24:56,786: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8da66ddf-df14-4a27-ae54-afe9dc2d15af, queue=dev-awx-0)
2021-06-06 11:24:56,823 DEBUG awx.main.dispatch delivered 8da66ddf-df14-4a27-ae54-afe9dc2d15af to worker[192] qsize 0
2021-06-06 11:24:56,830 DEBUG awx.main.dispatch task 8da66ddf-df14-4a27-ae54-afe9dc2d15af starting awx.main.tasks.awx_k8s_reaper(*[])
2021-06-06 11:24:56,849 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139838726587968_3" does not exist
2021-06-06 11:24:56,856 DEBUG awx.main.dispatch task 8da66ddf-df14-4a27-ae54-afe9dc2d15af is finished
2021-06-06 11:24:57,015 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(07580c0e-c326-4dab-a7ef-d9a93783eb45, queue=awx_private_queue)
[2021-06-06 11:24:57,015: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(07580c0e-c326-4dab-a7ef-d9a93783eb45, queue=awx_private_queue)
2021-06-06 11:24:57,052 DEBUG awx.main.dispatch delivered 07580c0e-c326-4dab-a7ef-d9a93783eb45 to worker[192] qsize 0
2021-06-06 11:24:57,075 DEBUG awx.main.dispatch task 07580c0e-c326-4dab-a7ef-d9a93783eb45 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:24:57,095 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:24:57,097 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:24:57,111 DEBUG awx.main.dispatch task 07580c0e-c326-4dab-a7ef-d9a93783eb45 is finished
RESULT 2
OKREADY
2021-06-06 11:25:17,025 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(94f60af2-3a42-418d-8f55-9edba7fbd7b0, queue=awx_private_queue)
[2021-06-06 11:25:17,025: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(94f60af2-3a42-418d-8f55-9edba7fbd7b0, queue=awx_private_queue)
2021-06-06 11:25:17,059 DEBUG awx.main.dispatch delivered 94f60af2-3a42-418d-8f55-9edba7fbd7b0 to worker[181] qsize 0
2021-06-06 11:25:17,063 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b3523166-b898-4f64-8f7b-2e30fdffd9ec, queue=awx_private_queue)
[2021-06-06 11:25:17,063: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b3523166-b898-4f64-8f7b-2e30fdffd9ec, queue=awx_private_queue)
2021-06-06 11:25:17,069 DEBUG awx.main.dispatch task 94f60af2-3a42-418d-8f55-9edba7fbd7b0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:25:17,090 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:25:17,092 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-06-06 11:25:17,107 DEBUG awx.main.dispatch task 94f60af2-3a42-418d-8f55-9edba7fbd7b0 is finished
2021-06-06 11:25:17,101 DEBUG awx.main.dispatch delivered b3523166-b898-4f64-8f7b-2e30fdffd9ec to worker[183] qsize 0
2021-06-06 11:25:17,117 DEBUG awx.main.dispatch task b3523166-b898-4f64-8f7b-2e30fdffd9ec starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-06-06 11:25:17,139 DEBUG awx.main.tasks Starting periodic scheduler
2021-06-06 11:25:17,141 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-06-06 11:25:17,145 DEBUG awx.main.dispatch task b3523166-b898-4f64-8f7b-2e30fdffd9ec is finished
2021-06-06 11:25:37,035 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ca408da4-7016-4baa-9bc7-5365f5020de0, queue=awx_private_queue)
[2021-06-06 11:25:37,035: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ca408da4-7016-4baa-9bc7-5365f5020de0, queue=awx_private_queue)
2021-06-06 11:25:37,067 DEBUG awx.main.dispatch delivered ca408da4-7016-4baa-9bc7-5365f5020de0 to worker[182] qsize 0
2021-06-06 11:25:37,074 DEBUG awx.main.dispatch task ca408da4-7016-4baa-9bc7-5365f5020de0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-06-06 11:25:37,087 DEBUG awx.main.scheduler Running Tower task manager.
2021-06-06 11:25:37,088 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File