By type
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"elapsed": 0,
"match_groupdict": {},
"match_groups": [],
"path": null,
"port": 5432,
"search_regex": null,
"state": "started"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"db": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
}
Default organization added.
Demo Credential, Inventory, and Job Template added.
(changed: True)
2021-04-13 10:51:26,348 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(66827e37-612e-45b5-96c0-6dcc8ecc338d, queue=awx_private_queue)
Successfully registered instance dev-awx-0
(changed: True)
2021-04-13 10:51:43,117 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(6ac418ac-a8c5-4a8e-9f48-44094e0f73c7, queue=awx_private_queue)
2021-04-13 10:51:43,325 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(72293f0f-96c6-4fb8-aec5-8e1584aaa602, queue=awx_private_queue)
Creating instance group tower
(changed: True)
2021-04-13 10:51:44,732 INFO RPC interface 'supervisor' initialized
2021-04-13 10:51:44,732 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2021-04-13 10:51:44,732 INFO supervisord started with pid 150
2021-04-13 10:51:45,736 INFO spawned: 'awx-config-watcher' with pid 153
2021-04-13 10:51:45,739 INFO spawned: 'dispatcher' with pid 154
2021-04-13 10:51:45,742 INFO spawned: 'callback-receiver' with pid 155
2021-04-13 10:51:45,745 INFO spawned: 'channels-worker' with pid 156
READY
2021-04-13 10:51:46,895 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-13 10:51:46,896 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-13 10:51:46,896 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-13 10:51:46,896 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-13 10:51:49,309 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:175
2021-04-13 10:51:49,309 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:175
2021-04-13 10:51:49,319 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-04-13 10:51:49,319 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-04-13 10:51:49,330 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2021-04-13 10:51:49,330 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2021-04-13 10:51:49,341 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2021-04-13 10:51:49,341 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2021-04-13 10:51:49,387 - INFO - runworker - Using single-threaded worker.
2021-04-13 10:51:49,387 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer)
2021-04-13 10:51:49,388 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive
2021-04-13 10:51:49,491 WARNING awx.main.dispatch scaling up worker pid:180
2021-04-13 10:51:49,505 WARNING awx.main.dispatch scaling up worker pid:181
2021-04-13 10:51:49,520 WARNING awx.main.dispatch scaling up worker pid:182
2021-04-13 10:51:49,530 WARNING awx.main.dispatch scaling up worker pid:183
2021-04-13 10:51:49,571 DEBUG awx.main.tasks Syncing Schedules
2021-04-13 10:51:49,645 ERROR celery.beat Removing corrupted schedule file '/var/lib/awx/beat.db': error(11, 'Resource temporarily unavailable')
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 485, in setup_schedule
self._store = self._open_schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 475, in _open_schedule
return self.persistence.open(self.schedule_filename, writeback=True)
File "/usr/lib64/python3.6/shelve.py", line 243, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/usr/lib64/python3.6/shelve.py", line 227, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
File "/usr/lib64/python3.6/dbm/__init__.py", line 94, in open
return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable
2021-04-13 10:51:50,006 DEBUG awx.main.tasks Waited 0.008561134338378906 seconds to obtain lock name: cluster_policy_lock
2021-04-13 10:51:50,029 DEBUG awx.main.tasks Total non-isolated instances:1 available for policy: 1
2021-04-13 10:51:50,029 DEBUG awx.main.tasks Policy percentage, adding Instances [1] to Group tower
2021-04-13 10:51:50,029 DEBUG awx.main.tasks Adding instances [1] to group tower
2021-04-13 10:51:50,055 DEBUG awx.main.tasks Cluster policy computation finished in 0.048501014709472656 seconds
2021-04-13 10:51:50,056 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 10:51:50,087 DEBUG awx.main.dispatch listening on ['awx_private_queue [direct]', 'dev-awx-0 [direct]', 'tower_broadcast_all_dev-awx-0 [fanout]']
RESULT 2
OKREADY
2021-04-13 10:52:10,932 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4cc41a54-bc37-41b7-8057-4e05dddcf039, queue=awx_private_queue)
[2021-04-13 10:52:10,932: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4cc41a54-bc37-41b7-8057-4e05dddcf039, queue=awx_private_queue)
2021-04-13 10:52:10,960 DEBUG awx.main.dispatch delivered 4cc41a54-bc37-41b7-8057-4e05dddcf039 to worker[182] qsize 0
2021-04-13 10:52:10,980 DEBUG awx.main.dispatch task 4cc41a54-bc37-41b7-8057-4e05dddcf039 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:52:10,988 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:52:11,035 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:52:11,088 DEBUG awx.main.dispatch task 4cc41a54-bc37-41b7-8057-4e05dddcf039 is finished
2021-04-13 10:52:19,835 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(66396df9-dc93-47a9-9ef5-5febef32116b, queue=awx_private_queue)
[2021-04-13 10:52:19,835: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(66396df9-dc93-47a9-9ef5-5febef32116b, queue=awx_private_queue)
2021-04-13 10:52:19,852 DEBUG awx.main.dispatch delivered 66396df9-dc93-47a9-9ef5-5febef32116b to worker[181] qsize 0
2021-04-13 10:52:19,857 DEBUG awx.main.dispatch task 66396df9-dc93-47a9-9ef5-5febef32116b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:52:19,881 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:52:19,917 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:52:19.912327+00:00
2021-04-13 10:52:19,959 DEBUG awx.main.dispatch task 66396df9-dc93-47a9-9ef5-5febef32116b is finished
2021-04-13 10:52:30,905 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5fd71caf-7b60-4a19-a7c8-4b3dde5461b4, queue=awx_private_queue)
[2021-04-13 10:52:30,905: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5fd71caf-7b60-4a19-a7c8-4b3dde5461b4, queue=awx_private_queue)
2021-04-13 10:52:30,922 DEBUG awx.main.dispatch delivered 5fd71caf-7b60-4a19-a7c8-4b3dde5461b4 to worker[181] qsize 0
2021-04-13 10:52:30,926 DEBUG awx.main.dispatch task 5fd71caf-7b60-4a19-a7c8-4b3dde5461b4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:52:30,927 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:52:30,958 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:52:31,012 DEBUG awx.main.dispatch task 5fd71caf-7b60-4a19-a7c8-4b3dde5461b4 is finished
2021-04-13 10:52:49,850 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(35ddcda7-984a-470b-a4f5-fd2eccada97a, queue=dev-awx-0)
[2021-04-13 10:52:49,850: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(35ddcda7-984a-470b-a4f5-fd2eccada97a, queue=dev-awx-0)
2021-04-13 10:52:50,110 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(caaf206a-16ab-4ddc-91af-2290f3694999, queue=dev-awx-0)
[2021-04-13 10:52:50,110: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(caaf206a-16ab-4ddc-91af-2290f3694999, queue=dev-awx-0)
2021-04-13 10:52:50,113 DEBUG awx.main.dispatch delivered 35ddcda7-984a-470b-a4f5-fd2eccada97a to worker[183] qsize 0
2021-04-13 10:52:50,122 DEBUG awx.main.dispatch task 35ddcda7-984a-470b-a4f5-fd2eccada97a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 10:52:50,140 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ab98d54f-c491-4fbc-be52-4a00f628fd84, queue=awx_private_queue)
[2021-04-13 10:52:50,140: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ab98d54f-c491-4fbc-be52-4a00f628fd84, queue=awx_private_queue)
2021-04-13 10:52:50,150 DEBUG awx.main.dispatch delivered caaf206a-16ab-4ddc-91af-2290f3694999 to worker[181] qsize 0
2021-04-13 10:52:50,154 DEBUG awx.main.dispatch task caaf206a-16ab-4ddc-91af-2290f3694999 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 10:52:50,159 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 10:52:50,158 DEBUG awx.main.dispatch delivered ab98d54f-c491-4fbc-be52-4a00f628fd84 to worker[180] qsize 0
2021-04-13 10:52:50,162 DEBUG awx.main.dispatch task 35ddcda7-984a-470b-a4f5-fd2eccada97a is finished
2021-04-13 10:52:50,161 DEBUG awx.main.dispatch task ab98d54f-c491-4fbc-be52-4a00f628fd84 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:52:50,176 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:52:50,181 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:52:19.881615+00:00
2021-04-13 10:52:50,195 DEBUG awx.main.dispatch task caaf206a-16ab-4ddc-91af-2290f3694999 is finished
2021-04-13 10:52:50,212 DEBUG awx.main.dispatch task ab98d54f-c491-4fbc-be52-4a00f628fd84 is finished
2021-04-13 10:52:50,905 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f9f28787-6aec-4927-9fe0-e71a32c71572, queue=awx_private_queue)
[2021-04-13 10:52:50,905: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f9f28787-6aec-4927-9fe0-e71a32c71572, queue=awx_private_queue)
2021-04-13 10:52:50,920 DEBUG awx.main.dispatch delivered f9f28787-6aec-4927-9fe0-e71a32c71572 to worker[182] qsize 0
2021-04-13 10:52:50,923 DEBUG awx.main.dispatch task f9f28787-6aec-4927-9fe0-e71a32c71572 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:52:50,927 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:52:50,945 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:52:50,971 DEBUG awx.main.dispatch task f9f28787-6aec-4927-9fe0-e71a32c71572 is finished
RESULT 2
OKREADY
2021-04-13 10:53:10,916 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(026b82fe-ba80-461c-b9b6-c12a45ea3b43, queue=awx_private_queue)
[2021-04-13 10:53:10,916: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(026b82fe-ba80-461c-b9b6-c12a45ea3b43, queue=awx_private_queue)
2021-04-13 10:53:10,939 DEBUG awx.main.dispatch delivered 026b82fe-ba80-461c-b9b6-c12a45ea3b43 to worker[182] qsize 0
2021-04-13 10:53:10,941 DEBUG awx.main.dispatch task 026b82fe-ba80-461c-b9b6-c12a45ea3b43 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:53:10,943 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:53:10,973 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:53:11,020 DEBUG awx.main.dispatch task 026b82fe-ba80-461c-b9b6-c12a45ea3b43 is finished
2021-04-13 10:53:20,140 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2d057193-797e-4278-ada3-fa4bb3fee04f, queue=awx_private_queue)
[2021-04-13 10:53:20,140: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2d057193-797e-4278-ada3-fa4bb3fee04f, queue=awx_private_queue)
2021-04-13 10:53:20,158 DEBUG awx.main.dispatch delivered 2d057193-797e-4278-ada3-fa4bb3fee04f to worker[183] qsize 0
2021-04-13 10:53:20,161 DEBUG awx.main.dispatch task 2d057193-797e-4278-ada3-fa4bb3fee04f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:53:20,171 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:53:20,174 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:52:50.176677+00:00
2021-04-13 10:53:20,200 DEBUG awx.main.dispatch task 2d057193-797e-4278-ada3-fa4bb3fee04f is finished
2021-04-13 10:53:30,916 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b1e7e996-8022-4d19-8786-ee052c2fff10, queue=awx_private_queue)
[2021-04-13 10:53:30,916: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b1e7e996-8022-4d19-8786-ee052c2fff10, queue=awx_private_queue)
2021-04-13 10:53:30,931 DEBUG awx.main.dispatch delivered b1e7e996-8022-4d19-8786-ee052c2fff10 to worker[180] qsize 0
2021-04-13 10:53:30,935 DEBUG awx.main.dispatch task b1e7e996-8022-4d19-8786-ee052c2fff10 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:53:30,937 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:53:30,958 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:53:31,008 DEBUG awx.main.dispatch task b1e7e996-8022-4d19-8786-ee052c2fff10 is finished
2021-04-13 10:53:49,861 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ea5754a9-9f7e-4dc6-8493-c6033180e5ff, queue=dev-awx-0)
[2021-04-13 10:53:49,861: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ea5754a9-9f7e-4dc6-8493-c6033180e5ff, queue=dev-awx-0)
2021-04-13 10:53:49,915 DEBUG awx.main.dispatch delivered ea5754a9-9f7e-4dc6-8493-c6033180e5ff to worker[182] qsize 0
2021-04-13 10:53:49,921 DEBUG awx.main.dispatch task ea5754a9-9f7e-4dc6-8493-c6033180e5ff starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 10:53:49,949 DEBUG awx.main.dispatch task ea5754a9-9f7e-4dc6-8493-c6033180e5ff is finished
2021-04-13 10:53:50,110 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(7345c86e-932a-4188-a59b-122f02932503, queue=dev-awx-0)
[2021-04-13 10:53:50,110: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(7345c86e-932a-4188-a59b-122f02932503, queue=dev-awx-0)
2021-04-13 10:53:50,140 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(44cbb1d0-560b-4c91-a5c6-1db27a99ee29, queue=awx_private_queue)
[2021-04-13 10:53:50,140: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(44cbb1d0-560b-4c91-a5c6-1db27a99ee29, queue=awx_private_queue)
2021-04-13 10:53:50,155 DEBUG awx.main.dispatch delivered 44cbb1d0-560b-4c91-a5c6-1db27a99ee29 to worker[181] qsize 0
2021-04-13 10:53:50,343 DEBUG awx.main.dispatch task 44cbb1d0-560b-4c91-a5c6-1db27a99ee29 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:53:50,359 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:53:50,363 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:53:20.171543+00:00
2021-04-13 10:53:50,385 DEBUG awx.main.dispatch task 44cbb1d0-560b-4c91-a5c6-1db27a99ee29 is finished
2021-04-13 10:53:50,916 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ca7bfb6a-0efb-456e-bf45-d5301baadb73, queue=awx_private_queue)
[2021-04-13 10:53:50,916: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ca7bfb6a-0efb-456e-bf45-d5301baadb73, queue=awx_private_queue)
2021-04-13 10:53:50,970 DEBUG awx.main.dispatch delivered 7345c86e-932a-4188-a59b-122f02932503 to worker[180] qsize 0
2021-04-13 10:53:50,974 DEBUG awx.main.dispatch task 7345c86e-932a-4188-a59b-122f02932503 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 10:53:50,979 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 10:53:51,011 DEBUG awx.main.dispatch delivered ca7bfb6a-0efb-456e-bf45-d5301baadb73 to worker[181] qsize 0
2021-04-13 10:53:51,014 DEBUG awx.main.dispatch task ca7bfb6a-0efb-456e-bf45-d5301baadb73 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:53:51,015 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:53:51,015 DEBUG awx.main.dispatch task 7345c86e-932a-4188-a59b-122f02932503 is finished
2021-04-13 10:53:51,046 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:53:51,104 DEBUG awx.main.dispatch task ca7bfb6a-0efb-456e-bf45-d5301baadb73 is finished
RESULT 2
OKREADY
2021-04-13 10:54:10,927 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(dbcb0212-d035-480e-8e04-71e38df7ec8e, queue=awx_private_queue)
[2021-04-13 10:54:10,927: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(dbcb0212-d035-480e-8e04-71e38df7ec8e, queue=awx_private_queue)
2021-04-13 10:54:10,943 DEBUG awx.main.dispatch delivered dbcb0212-d035-480e-8e04-71e38df7ec8e to worker[180] qsize 0
2021-04-13 10:54:10,945 DEBUG awx.main.dispatch task dbcb0212-d035-480e-8e04-71e38df7ec8e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:54:10,946 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:54:10,980 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:54:11,011 DEBUG awx.main.dispatch task dbcb0212-d035-480e-8e04-71e38df7ec8e is finished
2021-04-13 10:54:20,140 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0b67ee89-5c62-4603-aabe-3fa2b14a2a22, queue=awx_private_queue)
[2021-04-13 10:54:20,140: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0b67ee89-5c62-4603-aabe-3fa2b14a2a22, queue=awx_private_queue)
2021-04-13 10:54:20,154 DEBUG awx.main.dispatch delivered 0b67ee89-5c62-4603-aabe-3fa2b14a2a22 to worker[180] qsize 0
2021-04-13 10:54:20,156 DEBUG awx.main.dispatch task 0b67ee89-5c62-4603-aabe-3fa2b14a2a22 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:54:20,171 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:54:20,174 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:53:50.359216+00:00
2021-04-13 10:54:20,191 DEBUG awx.main.dispatch task 0b67ee89-5c62-4603-aabe-3fa2b14a2a22 is finished
2021-04-13 10:54:30,927 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2b505e03-638a-4e49-9d17-efc32b9985a2, queue=awx_private_queue)
[2021-04-13 10:54:30,927: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2b505e03-638a-4e49-9d17-efc32b9985a2, queue=awx_private_queue)
2021-04-13 10:54:30,958 DEBUG awx.main.dispatch delivered 2b505e03-638a-4e49-9d17-efc32b9985a2 to worker[182] qsize 0
2021-04-13 10:54:30,960 DEBUG awx.main.dispatch task 2b505e03-638a-4e49-9d17-efc32b9985a2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:54:30,963 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:54:30,996 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:54:31,033 DEBUG awx.main.dispatch task 2b505e03-638a-4e49-9d17-efc32b9985a2 is finished
2021-04-13 10:54:49,870 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f65f76c4-cfce-4d45-95bc-c3c453a7cc77, queue=dev-awx-0)
[2021-04-13 10:54:49,870: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f65f76c4-cfce-4d45-95bc-c3c453a7cc77, queue=dev-awx-0)
2021-04-13 10:54:49,913 DEBUG awx.main.dispatch delivered f65f76c4-cfce-4d45-95bc-c3c453a7cc77 to worker[180] qsize 0
2021-04-13 10:54:49,920 DEBUG awx.main.dispatch task f65f76c4-cfce-4d45-95bc-c3c453a7cc77 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 10:54:50,148 DEBUG awx.main.dispatch task f65f76c4-cfce-4d45-95bc-c3c453a7cc77 is finished
2021-04-13 10:54:50,110 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d2c88395-6b0a-48c1-90fd-dac7aecc6ae4, queue=dev-awx-0)
[2021-04-13 10:54:50,110: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d2c88395-6b0a-48c1-90fd-dac7aecc6ae4, queue=dev-awx-0)
2021-04-13 10:54:50,175 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(58cdb02a-a2e8-4eae-9eab-e72ca0da0a85, queue=awx_private_queue)
[2021-04-13 10:54:50,175: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(58cdb02a-a2e8-4eae-9eab-e72ca0da0a85, queue=awx_private_queue)
2021-04-13 10:54:50,190 DEBUG awx.main.dispatch delivered 58cdb02a-a2e8-4eae-9eab-e72ca0da0a85 to worker[182] qsize 0
2021-04-13 10:54:50,390 DEBUG awx.main.dispatch task 58cdb02a-a2e8-4eae-9eab-e72ca0da0a85 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:54:50,424 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:54:50,429 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:54:20.171376+00:00
2021-04-13 10:54:50,504 DEBUG awx.main.dispatch task 58cdb02a-a2e8-4eae-9eab-e72ca0da0a85 is finished
2021-04-13 10:54:50,926 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c20e334a-107d-4224-9f31-d765d7f50631, queue=awx_private_queue)
[2021-04-13 10:54:50,926: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c20e334a-107d-4224-9f31-d765d7f50631, queue=awx_private_queue)
2021-04-13 10:54:51,133 DEBUG awx.main.dispatch delivered d2c88395-6b0a-48c1-90fd-dac7aecc6ae4 to worker[183] qsize 0
2021-04-13 10:54:51,137 DEBUG awx.main.dispatch task d2c88395-6b0a-48c1-90fd-dac7aecc6ae4 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 10:54:51,142 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 10:54:51,189 DEBUG awx.main.dispatch delivered c20e334a-107d-4224-9f31-d765d7f50631 to worker[180] qsize 0
2021-04-13 10:54:51,191 DEBUG awx.main.dispatch task c20e334a-107d-4224-9f31-d765d7f50631 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:54:51,193 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:54:51,323 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:54:51,355 DEBUG awx.main.dispatch task c20e334a-107d-4224-9f31-d765d7f50631 is finished
2021-04-13 10:54:51,718 DEBUG awx.main.dispatch task d2c88395-6b0a-48c1-90fd-dac7aecc6ae4 is finished
RESULT 2
OKREADY
2021-04-13 10:55:11,187 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b61543ef-1e72-462b-858f-c134d5daf2bd, queue=awx_private_queue)
[2021-04-13 10:55:11,187: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b61543ef-1e72-462b-858f-c134d5daf2bd, queue=awx_private_queue)
2021-04-13 10:55:11,204 DEBUG awx.main.dispatch delivered b61543ef-1e72-462b-858f-c134d5daf2bd to worker[181] qsize 0
2021-04-13 10:55:11,208 DEBUG awx.main.dispatch task b61543ef-1e72-462b-858f-c134d5daf2bd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:55:11,212 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:55:11,248 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:55:11,284 DEBUG awx.main.dispatch task b61543ef-1e72-462b-858f-c134d5daf2bd is finished
2021-04-13 10:55:20,175 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a35b2221-8d0f-4f3f-b877-54cef33c788c, queue=awx_private_queue)
[2021-04-13 10:55:20,175: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a35b2221-8d0f-4f3f-b877-54cef33c788c, queue=awx_private_queue)
2021-04-13 10:55:20,189 DEBUG awx.main.dispatch delivered a35b2221-8d0f-4f3f-b877-54cef33c788c to worker[180] qsize 0
2021-04-13 10:55:20,191 DEBUG awx.main.dispatch task a35b2221-8d0f-4f3f-b877-54cef33c788c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:55:20,232 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:55:20,235 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:54:50.424512+00:00
2021-04-13 10:55:20,250 DEBUG awx.main.dispatch task a35b2221-8d0f-4f3f-b877-54cef33c788c is finished
2021-04-13 10:55:31,193 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(90c9b162-37e6-4037-a84f-fa0b77722b06, queue=awx_private_queue)
[2021-04-13 10:55:31,193: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(90c9b162-37e6-4037-a84f-fa0b77722b06, queue=awx_private_queue)
2021-04-13 10:55:31,215 DEBUG awx.main.dispatch delivered 90c9b162-37e6-4037-a84f-fa0b77722b06 to worker[180] qsize 0
2021-04-13 10:55:31,222 DEBUG awx.main.dispatch task 90c9b162-37e6-4037-a84f-fa0b77722b06 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:55:31,226 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:55:31,253 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:55:31,291 DEBUG awx.main.dispatch task 90c9b162-37e6-4037-a84f-fa0b77722b06 is finished
2021-04-13 10:55:49,867 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(161ac99f-1a69-48ad-884f-ed0d2a54e49a, queue=dev-awx-0)
[2021-04-13 10:55:49,867: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(161ac99f-1a69-48ad-884f-ed0d2a54e49a, queue=dev-awx-0)
2021-04-13 10:55:49,888 DEBUG awx.main.dispatch delivered 161ac99f-1a69-48ad-884f-ed0d2a54e49a to worker[180] qsize 0
2021-04-13 10:55:49,892 DEBUG awx.main.dispatch task 161ac99f-1a69-48ad-884f-ed0d2a54e49a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 10:55:49,914 DEBUG awx.main.dispatch task 161ac99f-1a69-48ad-884f-ed0d2a54e49a is finished
2021-04-13 10:55:50,110 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5e47a959-95a0-42d4-8f85-cc37bbd84410, queue=dev-awx-0)
[2021-04-13 10:55:50,110: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5e47a959-95a0-42d4-8f85-cc37bbd84410, queue=dev-awx-0)
2021-04-13 10:55:50,164 DEBUG awx.main.dispatch delivered 5e47a959-95a0-42d4-8f85-cc37bbd84410 to worker[180] qsize 0
2021-04-13 10:55:50,165 DEBUG awx.main.dispatch task 5e47a959-95a0-42d4-8f85-cc37bbd84410 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 10:55:50,167 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 10:55:50,176 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9c8b0ff8-23b1-41c5-b301-3554e4aca631, queue=awx_private_queue)
[2021-04-13 10:55:50,176: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9c8b0ff8-23b1-41c5-b301-3554e4aca631, queue=awx_private_queue)
2021-04-13 10:55:50,191 DEBUG awx.main.dispatch task 5e47a959-95a0-42d4-8f85-cc37bbd84410 is finished
2021-04-13 10:55:50,214 DEBUG awx.main.dispatch delivered 9c8b0ff8-23b1-41c5-b301-3554e4aca631 to worker[183] qsize 0
2021-04-13 10:55:50,403 DEBUG awx.main.dispatch task 9c8b0ff8-23b1-41c5-b301-3554e4aca631 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:55:50,415 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:55:50,420 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:55:20.232539+00:00
2021-04-13 10:55:50,435 DEBUG awx.main.dispatch task 9c8b0ff8-23b1-41c5-b301-3554e4aca631 is finished
2021-04-13 10:55:51,190 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fb69847d-fae9-4842-9133-7624e4085c78, queue=awx_private_queue)
[2021-04-13 10:55:51,190: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fb69847d-fae9-4842-9133-7624e4085c78, queue=awx_private_queue)
2021-04-13 10:55:51,205 DEBUG awx.main.dispatch delivered fb69847d-fae9-4842-9133-7624e4085c78 to worker[183] qsize 0
2021-04-13 10:55:51,209 DEBUG awx.main.dispatch task fb69847d-fae9-4842-9133-7624e4085c78 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:55:51,210 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:55:51,231 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:55:51,268 DEBUG awx.main.dispatch task fb69847d-fae9-4842-9133-7624e4085c78 is finished
RESULT 2
OKREADY
2021-04-13 10:56:11,200 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(25311f9f-39cf-49b4-9e38-f8ad2531e5fb, queue=awx_private_queue)
[2021-04-13 10:56:11,200: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(25311f9f-39cf-49b4-9e38-f8ad2531e5fb, queue=awx_private_queue)
2021-04-13 10:56:11,217 DEBUG awx.main.dispatch delivered 25311f9f-39cf-49b4-9e38-f8ad2531e5fb to worker[183] qsize 0
2021-04-13 10:56:11,219 DEBUG awx.main.dispatch task 25311f9f-39cf-49b4-9e38-f8ad2531e5fb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:56:11,221 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:56:11,252 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:56:11,291 DEBUG awx.main.dispatch task 25311f9f-39cf-49b4-9e38-f8ad2531e5fb is finished
2021-04-13 10:56:20,175 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(cdcaaf11-ed45-48cc-acc7-5ee25c26121d, queue=awx_private_queue)
[2021-04-13 10:56:20,175: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(cdcaaf11-ed45-48cc-acc7-5ee25c26121d, queue=awx_private_queue)
2021-04-13 10:56:20,193 DEBUG awx.main.dispatch delivered cdcaaf11-ed45-48cc-acc7-5ee25c26121d to worker[181] qsize 0
2021-04-13 10:56:20,197 DEBUG awx.main.dispatch task cdcaaf11-ed45-48cc-acc7-5ee25c26121d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:56:20,213 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:56:20,218 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:55:50.415667+00:00
2021-04-13 10:56:20,240 DEBUG awx.main.dispatch task cdcaaf11-ed45-48cc-acc7-5ee25c26121d is finished
2021-04-13 10:56:31,201 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(01cc1fd9-2760-4c28-ac84-21f48e9ff287, queue=awx_private_queue)
[2021-04-13 10:56:31,201: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(01cc1fd9-2760-4c28-ac84-21f48e9ff287, queue=awx_private_queue)
2021-04-13 10:56:31,217 DEBUG awx.main.dispatch delivered 01cc1fd9-2760-4c28-ac84-21f48e9ff287 to worker[182] qsize 0
2021-04-13 10:56:31,221 DEBUG awx.main.dispatch task 01cc1fd9-2760-4c28-ac84-21f48e9ff287 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:56:31,226 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:56:31,252 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:56:31,287 DEBUG awx.main.dispatch task 01cc1fd9-2760-4c28-ac84-21f48e9ff287 is finished
2021-04-13 10:56:49,878 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a40a3faf-74d3-4b55-9de4-3c513cc13d90, queue=dev-awx-0)
[2021-04-13 10:56:49,878: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a40a3faf-74d3-4b55-9de4-3c513cc13d90, queue=dev-awx-0)
2021-04-13 10:56:49,904 DEBUG awx.main.dispatch delivered a40a3faf-74d3-4b55-9de4-3c513cc13d90 to worker[182] qsize 0
2021-04-13 10:56:49,911 DEBUG awx.main.dispatch task a40a3faf-74d3-4b55-9de4-3c513cc13d90 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 10:56:49,930 DEBUG awx.main.dispatch task a40a3faf-74d3-4b55-9de4-3c513cc13d90 is finished
2021-04-13 10:56:50,111 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(24fa8561-b84e-4b27-8b4f-443fe5e14247, queue=dev-awx-0)
[2021-04-13 10:56:50,111: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(24fa8561-b84e-4b27-8b4f-443fe5e14247, queue=dev-awx-0)
2021-04-13 10:56:50,175 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(28f2658a-a201-4bdf-a152-d919aaef8691, queue=awx_private_queue)
[2021-04-13 10:56:50,175: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(28f2658a-a201-4bdf-a152-d919aaef8691, queue=awx_private_queue)
2021-04-13 10:56:50,189 DEBUG awx.main.dispatch delivered 28f2658a-a201-4bdf-a152-d919aaef8691 to worker[181] qsize 0
2021-04-13 10:56:50,193 DEBUG awx.main.dispatch task 28f2658a-a201-4bdf-a152-d919aaef8691 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:56:50,205 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:56:50,209 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:56:20.213887+00:00
2021-04-13 10:56:50,223 DEBUG awx.main.dispatch task 28f2658a-a201-4bdf-a152-d919aaef8691 is finished
2021-04-13 10:56:51,202 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bf6ea0d0-3e6f-4234-b4b6-88ee2ed07b62, queue=awx_private_queue)
[2021-04-13 10:56:51,202: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bf6ea0d0-3e6f-4234-b4b6-88ee2ed07b62, queue=awx_private_queue)
2021-04-13 10:56:51,515 DEBUG awx.main.dispatch delivered 24fa8561-b84e-4b27-8b4f-443fe5e14247 to worker[180] qsize 0
2021-04-13 10:56:51,698 DEBUG awx.main.dispatch delivered bf6ea0d0-3e6f-4234-b4b6-88ee2ed07b62 to worker[181] qsize 0
2021-04-13 10:56:51,700 DEBUG awx.main.dispatch task bf6ea0d0-3e6f-4234-b4b6-88ee2ed07b62 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:56:51,702 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:56:51,700 DEBUG awx.main.dispatch task 24fa8561-b84e-4b27-8b4f-443fe5e14247 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 10:56:51,706 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 10:56:51,724 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:56:51,736 DEBUG awx.main.dispatch task 24fa8561-b84e-4b27-8b4f-443fe5e14247 is finished
2021-04-13 10:56:51,756 DEBUG awx.main.dispatch task bf6ea0d0-3e6f-4234-b4b6-88ee2ed07b62 is finished
RESULT 2
OKREADY
2021-04-13 10:57:11,212 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(928084e0-e270-47bc-aa9c-312ab0614b3c, queue=awx_private_queue)
[2021-04-13 10:57:11,212: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(928084e0-e270-47bc-aa9c-312ab0614b3c, queue=awx_private_queue)
2021-04-13 10:57:11,244 DEBUG awx.main.dispatch delivered 928084e0-e270-47bc-aa9c-312ab0614b3c to worker[183] qsize 0
2021-04-13 10:57:11,248 DEBUG awx.main.dispatch task 928084e0-e270-47bc-aa9c-312ab0614b3c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:57:11,255 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:57:11,282 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:57:11,316 DEBUG awx.main.dispatch task 928084e0-e270-47bc-aa9c-312ab0614b3c is finished
2021-04-13 10:57:20,176 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f88939c4-77ba-43b3-bb21-09286dec73ae, queue=awx_private_queue)
[2021-04-13 10:57:20,176: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f88939c4-77ba-43b3-bb21-09286dec73ae, queue=awx_private_queue)
2021-04-13 10:57:20,198 DEBUG awx.main.dispatch delivered f88939c4-77ba-43b3-bb21-09286dec73ae to worker[181] qsize 0
2021-04-13 10:57:20,200 DEBUG awx.main.dispatch task f88939c4-77ba-43b3-bb21-09286dec73ae starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:57:20,214 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:57:20,218 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:56:50.205889+00:00
2021-04-13 10:57:20,229 DEBUG awx.main.dispatch task f88939c4-77ba-43b3-bb21-09286dec73ae is finished
2021-04-13 10:57:31,212 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e06098ff-3250-44da-8702-1b61b67acaa8, queue=awx_private_queue)
[2021-04-13 10:57:31,212: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e06098ff-3250-44da-8702-1b61b67acaa8, queue=awx_private_queue)
2021-04-13 10:57:31,228 DEBUG awx.main.dispatch delivered e06098ff-3250-44da-8702-1b61b67acaa8 to worker[182] qsize 0
2021-04-13 10:57:31,231 DEBUG awx.main.dispatch task e06098ff-3250-44da-8702-1b61b67acaa8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:57:31,233 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:57:31,253 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:57:31,287 DEBUG awx.main.dispatch task e06098ff-3250-44da-8702-1b61b67acaa8 is finished
2021-04-13 10:57:49,890 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c8044044-e2fe-43fe-947e-bab6808a67dd, queue=dev-awx-0)
[2021-04-13 10:57:49,890: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c8044044-e2fe-43fe-947e-bab6808a67dd, queue=dev-awx-0)
2021-04-13 10:57:49,920 DEBUG awx.main.dispatch delivered c8044044-e2fe-43fe-947e-bab6808a67dd to worker[181] qsize 0
2021-04-13 10:57:49,928 DEBUG awx.main.dispatch task c8044044-e2fe-43fe-947e-bab6808a67dd starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 10:57:49,966 DEBUG awx.main.dispatch task c8044044-e2fe-43fe-947e-bab6808a67dd is finished
2021-04-13 10:57:50,110 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(72c9dc47-e0d7-47b0-8fe5-7b4f34c01149, queue=dev-awx-0)
[2021-04-13 10:57:50,110: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(72c9dc47-e0d7-47b0-8fe5-7b4f34c01149, queue=dev-awx-0)
2021-04-13 10:57:50,172 DEBUG awx.main.dispatch delivered 72c9dc47-e0d7-47b0-8fe5-7b4f34c01149 to worker[180] qsize 0
2021-04-13 10:57:50,175 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dfcd2ee2-0c37-487e-9e73-d47711793b1e, queue=awx_private_queue)
[2021-04-13 10:57:50,175: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dfcd2ee2-0c37-487e-9e73-d47711793b1e, queue=awx_private_queue)
2021-04-13 10:57:50,177 DEBUG awx.main.dispatch task 72c9dc47-e0d7-47b0-8fe5-7b4f34c01149 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 10:57:50,183 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 10:57:50,192 DEBUG awx.main.dispatch delivered dfcd2ee2-0c37-487e-9e73-d47711793b1e to worker[183] qsize 0
2021-04-13 10:57:50,195 DEBUG awx.main.dispatch task dfcd2ee2-0c37-487e-9e73-d47711793b1e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:57:50,318 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:57:50,323 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:57:20.214913+00:00
2021-04-13 10:57:50,325 DEBUG awx.main.dispatch task 72c9dc47-e0d7-47b0-8fe5-7b4f34c01149 is finished
2021-04-13 10:57:50,339 DEBUG awx.main.dispatch task dfcd2ee2-0c37-487e-9e73-d47711793b1e is finished
2021-04-13 10:57:51,212 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9f681af3-57fe-4f6a-939c-5c909990395d, queue=awx_private_queue)
[2021-04-13 10:57:51,212: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9f681af3-57fe-4f6a-939c-5c909990395d, queue=awx_private_queue)
2021-04-13 10:57:51,618 DEBUG awx.main.dispatch delivered 9f681af3-57fe-4f6a-939c-5c909990395d to worker[181] qsize 0
2021-04-13 10:57:51,807 DEBUG awx.main.dispatch task 9f681af3-57fe-4f6a-939c-5c909990395d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:57:51,808 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:57:51,827 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:57:51,865 DEBUG awx.main.dispatch task 9f681af3-57fe-4f6a-939c-5c909990395d is finished
RESULT 2
OKREADY
2021-04-13 10:58:11,286 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ed1660a7-61e6-4aec-b5f9-6e2622558961, queue=awx_private_queue)
[2021-04-13 10:58:11,286: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ed1660a7-61e6-4aec-b5f9-6e2622558961, queue=awx_private_queue)
2021-04-13 10:58:11,312 DEBUG awx.main.dispatch delivered ed1660a7-61e6-4aec-b5f9-6e2622558961 to worker[180] qsize 0
2021-04-13 10:58:11,314 DEBUG awx.main.dispatch task ed1660a7-61e6-4aec-b5f9-6e2622558961 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:58:11,317 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:58:11,348 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:58:11,405 DEBUG awx.main.dispatch task ed1660a7-61e6-4aec-b5f9-6e2622558961 is finished
2021-04-13 10:58:20,175 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(864d2019-1be1-42a0-94b8-14e11ed7dfc4, queue=awx_private_queue)
[2021-04-13 10:58:20,175: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(864d2019-1be1-42a0-94b8-14e11ed7dfc4, queue=awx_private_queue)
2021-04-13 10:58:20,190 DEBUG awx.main.dispatch delivered 864d2019-1be1-42a0-94b8-14e11ed7dfc4 to worker[181] qsize 0
2021-04-13 10:58:20,192 DEBUG awx.main.dispatch task 864d2019-1be1-42a0-94b8-14e11ed7dfc4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:58:20,203 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:58:20,207 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:57:50.319073+00:00
2021-04-13 10:58:20,238 DEBUG awx.main.dispatch task 864d2019-1be1-42a0-94b8-14e11ed7dfc4 is finished
2021-04-13 10:58:31,286 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(353fe367-bcd7-435b-abc6-37f5e99991e9, queue=awx_private_queue)
[2021-04-13 10:58:31,286: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(353fe367-bcd7-435b-abc6-37f5e99991e9, queue=awx_private_queue)
2021-04-13 10:58:31,301 DEBUG awx.main.dispatch delivered 353fe367-bcd7-435b-abc6-37f5e99991e9 to worker[183] qsize 0
2021-04-13 10:58:31,303 DEBUG awx.main.dispatch task 353fe367-bcd7-435b-abc6-37f5e99991e9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:58:31,306 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:58:31,329 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:58:31,360 DEBUG awx.main.dispatch task 353fe367-bcd7-435b-abc6-37f5e99991e9 is finished
2021-04-13 10:58:49,898 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f138dc1f-e065-4b7c-a5ee-410cab7b3a48, queue=dev-awx-0)
[2021-04-13 10:58:49,898: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f138dc1f-e065-4b7c-a5ee-410cab7b3a48, queue=dev-awx-0)
2021-04-13 10:58:49,919 DEBUG awx.main.dispatch delivered f138dc1f-e065-4b7c-a5ee-410cab7b3a48 to worker[181] qsize 0
2021-04-13 10:58:49,924 DEBUG awx.main.dispatch task f138dc1f-e065-4b7c-a5ee-410cab7b3a48 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 10:58:49,940 DEBUG awx.main.dispatch task f138dc1f-e065-4b7c-a5ee-410cab7b3a48 is finished
2021-04-13 10:58:50,110 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(31efa58f-2a42-47f5-a4d0-8bccc89e6e8f, queue=dev-awx-0)
[2021-04-13 10:58:50,110: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(31efa58f-2a42-47f5-a4d0-8bccc89e6e8f, queue=dev-awx-0)
2021-04-13 10:58:50,176 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5923edd1-abc8-4c45-821d-fe28a14e7df1, queue=awx_private_queue)
[2021-04-13 10:58:50,176: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5923edd1-abc8-4c45-821d-fe28a14e7df1, queue=awx_private_queue)
2021-04-13 10:58:50,198 DEBUG awx.main.dispatch delivered 5923edd1-abc8-4c45-821d-fe28a14e7df1 to worker[181] qsize 0
2021-04-13 10:58:50,200 DEBUG awx.main.dispatch task 5923edd1-abc8-4c45-821d-fe28a14e7df1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:58:50,210 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:58:50,212 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:58:20.203197+00:00
2021-04-13 10:58:50,223 DEBUG awx.main.dispatch delivered 31efa58f-2a42-47f5-a4d0-8bccc89e6e8f to worker[182] qsize 0
2021-04-13 10:58:50,230 DEBUG awx.main.dispatch task 5923edd1-abc8-4c45-821d-fe28a14e7df1 is finished
2021-04-13 10:58:50,227 DEBUG awx.main.dispatch task 31efa58f-2a42-47f5-a4d0-8bccc89e6e8f starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 10:58:50,232 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 10:58:50,261 DEBUG awx.main.dispatch task 31efa58f-2a42-47f5-a4d0-8bccc89e6e8f is finished
2021-04-13 10:58:51,286 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0bab296f-f9c1-4e51-9fd2-a9c5e86ea46d, queue=awx_private_queue)
[2021-04-13 10:58:51,286: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0bab296f-f9c1-4e51-9fd2-a9c5e86ea46d, queue=awx_private_queue)
2021-04-13 10:58:51,310 DEBUG awx.main.dispatch delivered 0bab296f-f9c1-4e51-9fd2-a9c5e86ea46d to worker[183] qsize 0
2021-04-13 10:58:51,313 DEBUG awx.main.dispatch task 0bab296f-f9c1-4e51-9fd2-a9c5e86ea46d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:58:51,317 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:58:51,340 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:58:51,367 DEBUG awx.main.dispatch task 0bab296f-f9c1-4e51-9fd2-a9c5e86ea46d is finished
RESULT 2
OKREADY
2021-04-13 10:59:11,293 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8c6583b0-00dc-4338-bc6f-6ff19d9d0a4a, queue=awx_private_queue)
[2021-04-13 10:59:11,293: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8c6583b0-00dc-4338-bc6f-6ff19d9d0a4a, queue=awx_private_queue)
2021-04-13 10:59:11,321 DEBUG awx.main.dispatch delivered 8c6583b0-00dc-4338-bc6f-6ff19d9d0a4a to worker[182] qsize 0
2021-04-13 10:59:11,545 DEBUG awx.main.dispatch task 8c6583b0-00dc-4338-bc6f-6ff19d9d0a4a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:59:11,547 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:59:11,595 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:59:11,630 DEBUG awx.main.dispatch task 8c6583b0-00dc-4338-bc6f-6ff19d9d0a4a is finished
2021-04-13 10:59:20,176 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(530d144d-215a-44ef-95f1-32f7ff1b7ba6, queue=awx_private_queue)
[2021-04-13 10:59:20,176: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(530d144d-215a-44ef-95f1-32f7ff1b7ba6, queue=awx_private_queue)
2021-04-13 10:59:20,192 DEBUG awx.main.dispatch delivered 530d144d-215a-44ef-95f1-32f7ff1b7ba6 to worker[180] qsize 0
2021-04-13 10:59:20,196 DEBUG awx.main.dispatch task 530d144d-215a-44ef-95f1-32f7ff1b7ba6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:59:20,217 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:59:20,222 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:58:50.210345+00:00
2021-04-13 10:59:20,379 DEBUG awx.main.dispatch task 530d144d-215a-44ef-95f1-32f7ff1b7ba6 is finished
2021-04-13 10:59:31,289 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6f11dbc9-e1de-459d-b182-10e1e93f3550, queue=awx_private_queue)
[2021-04-13 10:59:31,289: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6f11dbc9-e1de-459d-b182-10e1e93f3550, queue=awx_private_queue)
2021-04-13 10:59:31,326 DEBUG awx.main.dispatch delivered 6f11dbc9-e1de-459d-b182-10e1e93f3550 to worker[181] qsize 0
2021-04-13 10:59:31,329 DEBUG awx.main.dispatch task 6f11dbc9-e1de-459d-b182-10e1e93f3550 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:59:31,332 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:59:31,364 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:59:31,403 DEBUG awx.main.dispatch task 6f11dbc9-e1de-459d-b182-10e1e93f3550 is finished
2021-04-13 10:59:49,900 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c28c86ac-dee2-4857-acd4-a097f9b240b7, queue=dev-awx-0)
[2021-04-13 10:59:49,900: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c28c86ac-dee2-4857-acd4-a097f9b240b7, queue=dev-awx-0)
2021-04-13 10:59:49,916 DEBUG awx.main.dispatch delivered c28c86ac-dee2-4857-acd4-a097f9b240b7 to worker[182] qsize 0
2021-04-13 10:59:49,918 DEBUG awx.main.dispatch task c28c86ac-dee2-4857-acd4-a097f9b240b7 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 10:59:49,938 DEBUG awx.main.dispatch task c28c86ac-dee2-4857-acd4-a097f9b240b7 is finished
2021-04-13 10:59:50,110 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(cd838051-6036-4031-aa72-be239c3466e9, queue=dev-awx-0)
[2021-04-13 10:59:50,110: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(cd838051-6036-4031-aa72-be239c3466e9, queue=dev-awx-0)
2021-04-13 10:59:50,159 DEBUG awx.main.dispatch delivered cd838051-6036-4031-aa72-be239c3466e9 to worker[180] qsize 0
2021-04-13 10:59:50,161 DEBUG awx.main.dispatch task cd838051-6036-4031-aa72-be239c3466e9 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 10:59:50,162 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 10:59:50,176 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9666140c-3967-4aa7-a420-65d1148d80a3, queue=awx_private_queue)
[2021-04-13 10:59:50,176: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9666140c-3967-4aa7-a420-65d1148d80a3, queue=awx_private_queue)
2021-04-13 10:59:50,187 DEBUG awx.main.dispatch task cd838051-6036-4031-aa72-be239c3466e9 is finished
2021-04-13 10:59:50,201 DEBUG awx.main.dispatch delivered 9666140c-3967-4aa7-a420-65d1148d80a3 to worker[182] qsize 0
2021-04-13 10:59:50,204 DEBUG awx.main.dispatch task 9666140c-3967-4aa7-a420-65d1148d80a3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 10:59:50,214 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 10:59:50,217 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:59:20.217666+00:00
2021-04-13 10:59:50,229 DEBUG awx.main.dispatch task 9666140c-3967-4aa7-a420-65d1148d80a3 is finished
2021-04-13 10:59:51,289 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(822312f8-385b-4083-baf6-ed890a1d1876, queue=awx_private_queue)
[2021-04-13 10:59:51,289: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(822312f8-385b-4083-baf6-ed890a1d1876, queue=awx_private_queue)
2021-04-13 10:59:51,304 DEBUG awx.main.dispatch delivered 822312f8-385b-4083-baf6-ed890a1d1876 to worker[180] qsize 0
2021-04-13 10:59:51,308 DEBUG awx.main.dispatch task 822312f8-385b-4083-baf6-ed890a1d1876 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 10:59:51,309 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 10:59:51,328 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 10:59:51,355 DEBUG awx.main.dispatch task 822312f8-385b-4083-baf6-ed890a1d1876 is finished
RESULT 2
OKREADY
2021-04-13 11:00:11,297 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6cae9bee-ef90-454a-ba04-1c0edde0b499, queue=awx_private_queue)
[2021-04-13 11:00:11,297: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6cae9bee-ef90-454a-ba04-1c0edde0b499, queue=awx_private_queue)
2021-04-13 11:00:11,312 DEBUG awx.main.dispatch delivered 6cae9bee-ef90-454a-ba04-1c0edde0b499 to worker[182] qsize 0
2021-04-13 11:00:11,314 DEBUG awx.main.dispatch task 6cae9bee-ef90-454a-ba04-1c0edde0b499 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:00:11,315 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:00:11,338 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:00:11,396 DEBUG awx.main.dispatch task 6cae9bee-ef90-454a-ba04-1c0edde0b499 is finished
2021-04-13 11:00:20,179 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a647df7f-b8cd-4be0-a52f-cdba734a99f6, queue=awx_private_queue)
[2021-04-13 11:00:20,179: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a647df7f-b8cd-4be0-a52f-cdba734a99f6, queue=awx_private_queue)
2021-04-13 11:00:20,228 DEBUG awx.main.dispatch delivered a647df7f-b8cd-4be0-a52f-cdba734a99f6 to worker[182] qsize 0
2021-04-13 11:00:20,428 DEBUG awx.main.dispatch task a647df7f-b8cd-4be0-a52f-cdba734a99f6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:00:20,441 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:00:20,449 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 10:59:50.214221+00:00
2021-04-13 11:00:20,466 DEBUG awx.main.dispatch task a647df7f-b8cd-4be0-a52f-cdba734a99f6 is finished
2021-04-13 11:00:31,302 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b05cec0f-d808-45ba-b0fe-f3bfc986b486, queue=awx_private_queue)
[2021-04-13 11:00:31,302: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b05cec0f-d808-45ba-b0fe-f3bfc986b486, queue=awx_private_queue)
2021-04-13 11:00:31,325 DEBUG awx.main.dispatch delivered b05cec0f-d808-45ba-b0fe-f3bfc986b486 to worker[182] qsize 0
2021-04-13 11:00:31,328 DEBUG awx.main.dispatch task b05cec0f-d808-45ba-b0fe-f3bfc986b486 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:00:31,330 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:00:31,348 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:00:31,377 DEBUG awx.main.dispatch task b05cec0f-d808-45ba-b0fe-f3bfc986b486 is finished
2021-04-13 11:00:49,908 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(635d2ecb-992d-40ca-83cc-24a14ed013bb, queue=dev-awx-0)
[2021-04-13 11:00:49,908: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(635d2ecb-992d-40ca-83cc-24a14ed013bb, queue=dev-awx-0)
2021-04-13 11:00:49,930 DEBUG awx.main.dispatch delivered 635d2ecb-992d-40ca-83cc-24a14ed013bb to worker[183] qsize 0
2021-04-13 11:00:49,934 DEBUG awx.main.dispatch task 635d2ecb-992d-40ca-83cc-24a14ed013bb starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:00:49,963 DEBUG awx.main.dispatch task 635d2ecb-992d-40ca-83cc-24a14ed013bb is finished
2021-04-13 11:00:50,110 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5eac59de-f905-45d1-af00-c6b9c4b1aa42, queue=dev-awx-0)
[2021-04-13 11:00:50,110: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5eac59de-f905-45d1-af00-c6b9c4b1aa42, queue=dev-awx-0)
2021-04-13 11:00:50,137 DEBUG awx.main.dispatch delivered 5eac59de-f905-45d1-af00-c6b9c4b1aa42 to worker[181] qsize 0
2021-04-13 11:00:50,140 DEBUG awx.main.dispatch task 5eac59de-f905-45d1-af00-c6b9c4b1aa42 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:00:50,145 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:00:50,176 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(30e4a052-545e-4b4f-964d-069f07ec0700, queue=awx_private_queue)
[2021-04-13 11:00:50,176: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(30e4a052-545e-4b4f-964d-069f07ec0700, queue=awx_private_queue)
2021-04-13 11:00:50,194 DEBUG awx.main.dispatch delivered 30e4a052-545e-4b4f-964d-069f07ec0700 to worker[180] qsize 0
2021-04-13 11:00:50,198 DEBUG awx.main.dispatch task 30e4a052-545e-4b4f-964d-069f07ec0700 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:00:50,453 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:00:50,459 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:00:20.442087+00:00
2021-04-13 11:00:50,483 DEBUG awx.main.dispatch task 30e4a052-545e-4b4f-964d-069f07ec0700 is finished
2021-04-13 11:00:50,483 DEBUG awx.main.dispatch task 5eac59de-f905-45d1-af00-c6b9c4b1aa42 is finished
2021-04-13 11:00:51,300 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(12596a77-f10a-427a-b1d7-940a4373d61b, queue=awx_private_queue)
[2021-04-13 11:00:51,300: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(12596a77-f10a-427a-b1d7-940a4373d61b, queue=awx_private_queue)
2021-04-13 11:00:51,315 DEBUG awx.main.dispatch delivered 12596a77-f10a-427a-b1d7-940a4373d61b to worker[180] qsize 0
2021-04-13 11:00:51,317 DEBUG awx.main.dispatch task 12596a77-f10a-427a-b1d7-940a4373d61b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:00:51,318 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:00:51,348 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:00:51,380 DEBUG awx.main.dispatch task 12596a77-f10a-427a-b1d7-940a4373d61b is finished
RESULT 2
OKREADY
2021-04-13 11:01:11,358 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5ce97709-7809-4947-bb64-0300ea7382e1, queue=awx_private_queue)
[2021-04-13 11:01:11,358: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5ce97709-7809-4947-bb64-0300ea7382e1, queue=awx_private_queue)
2021-04-13 11:01:11,375 DEBUG awx.main.dispatch delivered 5ce97709-7809-4947-bb64-0300ea7382e1 to worker[182] qsize 0
2021-04-13 11:01:11,376 DEBUG awx.main.dispatch task 5ce97709-7809-4947-bb64-0300ea7382e1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:01:11,378 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:01:11,995 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:01:12,030 DEBUG awx.main.dispatch task 5ce97709-7809-4947-bb64-0300ea7382e1 is finished
2021-04-13 11:01:20,176 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4b8b1763-8c77-4236-818f-cd630a923945, queue=awx_private_queue)
[2021-04-13 11:01:20,176: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4b8b1763-8c77-4236-818f-cd630a923945, queue=awx_private_queue)
2021-04-13 11:01:20,605 DEBUG awx.main.dispatch delivered 4b8b1763-8c77-4236-818f-cd630a923945 to worker[183] qsize 0
2021-04-13 11:01:20,790 DEBUG awx.main.dispatch task 4b8b1763-8c77-4236-818f-cd630a923945 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:01:20,807 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:01:20,824 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:00:50.453804+00:00
2021-04-13 11:01:20,842 DEBUG awx.main.dispatch task 4b8b1763-8c77-4236-818f-cd630a923945 is finished
2021-04-13 11:01:31,358 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2dfd3e95-dc77-480d-b47c-db050d54e632, queue=awx_private_queue)
[2021-04-13 11:01:31,358: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2dfd3e95-dc77-480d-b47c-db050d54e632, queue=awx_private_queue)
2021-04-13 11:01:31,372 DEBUG awx.main.dispatch delivered 2dfd3e95-dc77-480d-b47c-db050d54e632 to worker[183] qsize 0
2021-04-13 11:01:31,374 DEBUG awx.main.dispatch task 2dfd3e95-dc77-480d-b47c-db050d54e632 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:01:31,375 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:01:32,462 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:01:32,535 DEBUG awx.main.dispatch task 2dfd3e95-dc77-480d-b47c-db050d54e632 is finished
2021-04-13 11:01:49,835 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(2f5bb360-ef48-4dfe-b6a7-32960c4c4a0c, queue=dev-awx-0)
[2021-04-13 11:01:49,835: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(2f5bb360-ef48-4dfe-b6a7-32960c4c4a0c, queue=dev-awx-0)
2021-04-13 11:01:49,854 DEBUG awx.main.dispatch delivered 2f5bb360-ef48-4dfe-b6a7-32960c4c4a0c to worker[183] qsize 0
2021-04-13 11:01:49,856 DEBUG awx.main.dispatch task 2f5bb360-ef48-4dfe-b6a7-32960c4c4a0c starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-04-13 11:01:49,858 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-04-13 11:01:49,908 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(625fcf16-0c76-49d3-95cc-20bbc4dbe824, queue=dev-awx-0)
[2021-04-13 11:01:49,908: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(625fcf16-0c76-49d3-95cc-20bbc4dbe824, queue=dev-awx-0)
2021-04-13 11:01:49,925 DEBUG awx.main.dispatch delivered 625fcf16-0c76-49d3-95cc-20bbc4dbe824 to worker[181] qsize 0
2021-04-13 11:01:49,927 DEBUG awx.main.dispatch task 625fcf16-0c76-49d3-95cc-20bbc4dbe824 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:01:50,110 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e2e0e8e9-a52a-49f0-b211-a167f8951d1a, queue=dev-awx-0)
[2021-04-13 11:01:50,110: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e2e0e8e9-a52a-49f0-b211-a167f8951d1a, queue=dev-awx-0)
2021-04-13 11:01:50,177 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a96dfb58-c280-46f1-998f-00b2c32610d0, queue=awx_private_queue)
[2021-04-13 11:01:50,177: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a96dfb58-c280-46f1-998f-00b2c32610d0, queue=awx_private_queue)
2021-04-13 11:01:51,358 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c20dc8f8-ca27-4d41-a623-b30f7529aad0, queue=awx_private_queue)
[2021-04-13 11:01:51,358: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c20dc8f8-ca27-4d41-a623-b30f7529aad0, queue=awx_private_queue)
2021-04-13 11:01:57,475 DEBUG awx.main.dispatch task 2f5bb360-ef48-4dfe-b6a7-32960c4c4a0c is finished
RESULT 2
OKREADY
2021-04-13 11:02:11,371 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0a9c0c14-54b0-4565-b98f-d067f0f3fddc, queue=awx_private_queue)
[2021-04-13 11:02:11,371: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0a9c0c14-54b0-4565-b98f-d067f0f3fddc, queue=awx_private_queue)
2021-04-13 11:02:15,511 DEBUG awx.main.dispatch task 625fcf16-0c76-49d3-95cc-20bbc4dbe824 is finished
2021-04-13 11:02:15,541 DEBUG awx.main.dispatch delivered e2e0e8e9-a52a-49f0-b211-a167f8951d1a to worker[182] qsize 0
2021-04-13 11:02:15,546 DEBUG awx.main.dispatch delivered a96dfb58-c280-46f1-998f-00b2c32610d0 to worker[183] qsize 0
2021-04-13 11:02:15,548 DEBUG awx.main.dispatch delivered c20dc8f8-ca27-4d41-a623-b30f7529aad0 to worker[180] qsize 0
2021-04-13 11:02:15,550 DEBUG awx.main.dispatch delivered 0a9c0c14-54b0-4565-b98f-d067f0f3fddc to worker[181] qsize 0
2021-04-13 11:02:15,549 DEBUG awx.main.dispatch task a96dfb58-c280-46f1-998f-00b2c32610d0 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:02:15,551 DEBUG awx.main.dispatch task c20dc8f8-ca27-4d41-a623-b30f7529aad0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:02:15,555 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:02:15,547 DEBUG awx.main.dispatch task e2e0e8e9-a52a-49f0-b211-a167f8951d1a starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:02:15,561 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:02:15,559 DEBUG awx.main.dispatch task 0a9c0c14-54b0-4565-b98f-d067f0f3fddc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:02:15,563 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:02:15,867 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:02:15,874 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-13 11:02:15,875 DEBUG awx.main.dispatch task 0a9c0c14-54b0-4565-b98f-d067f0f3fddc is finished
2021-04-13 11:02:15,912 DEBUG awx.main.dispatch task e2e0e8e9-a52a-49f0-b211-a167f8951d1a is finished
2021-04-13 11:02:17,159 DEBUG awx.main.dispatch task c20dc8f8-ca27-4d41-a623-b30f7529aad0 is finished
2021-04-13 11:02:20,176 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e0cf7de4-b30d-4cf4-b18d-152f5dc76bd4, queue=awx_private_queue)
[2021-04-13 11:02:20,176: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e0cf7de4-b30d-4cf4-b18d-152f5dc76bd4, queue=awx_private_queue)
2021-04-13 11:02:20,195 DEBUG awx.main.dispatch delivered e0cf7de4-b30d-4cf4-b18d-152f5dc76bd4 to worker[180] qsize 0
2021-04-13 11:02:20,198 DEBUG awx.main.dispatch task e0cf7de4-b30d-4cf4-b18d-152f5dc76bd4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:02:36,964 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:02:42,120 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:01:20.808134+00:00
2021-04-13 11:02:31,369 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(996d40c7-fe74-4409-9845-fd730908e766, queue=awx_private_queue)
[2021-04-13 11:02:31,369: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(996d40c7-fe74-4409-9845-fd730908e766, queue=awx_private_queue)
2021-04-13 11:02:46,991 DEBUG awx.main.dispatch delivered 996d40c7-fe74-4409-9845-fd730908e766 to worker[181] qsize 0
2021-04-13 11:02:47,237 DEBUG awx.main.dispatch task 996d40c7-fe74-4409-9845-fd730908e766 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:02:47,238 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:02:46,560 DEBUG awx.main.dispatch task a96dfb58-c280-46f1-998f-00b2c32610d0 is finished
2021-04-13 11:02:48,459 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:02:48,468 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:02:36.964847+00:00
2021-04-13 11:02:49,910 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(990f4c40-110a-4064-bada-0e948d954dff, queue=dev-awx-0)
[2021-04-13 11:02:49,910: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(990f4c40-110a-4064-bada-0e948d954dff, queue=dev-awx-0)
2021-04-13 11:02:49,925 DEBUG awx.main.dispatch delivered 990f4c40-110a-4064-bada-0e948d954dff to worker[182] qsize 0
2021-04-13 11:02:49,927 DEBUG awx.main.dispatch task 990f4c40-110a-4064-bada-0e948d954dff starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:02:50,110 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(50c4f9a9-8b6a-421d-acec-33100700f7cf, queue=dev-awx-0)
[2021-04-13 11:02:50,110: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(50c4f9a9-8b6a-421d-acec-33100700f7cf, queue=dev-awx-0)
2021-04-13 11:02:50,176 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(38475c6d-8b8d-453e-8caa-0c8763d7dbb5, queue=awx_private_queue)
[2021-04-13 11:02:50,176: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(38475c6d-8b8d-453e-8caa-0c8763d7dbb5, queue=awx_private_queue)
2021-04-13 11:02:51,368 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c6724419-d9c9-4080-90e5-dcf49a5ce376, queue=awx_private_queue)
[2021-04-13 11:02:51,368: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c6724419-d9c9-4080-90e5-dcf49a5ce376, queue=awx_private_queue)
2021-04-13 11:02:58,711 DEBUG awx.main.dispatch task e0cf7de4-b30d-4cf4-b18d-152f5dc76bd4 is finished
2021-04-13 11:02:58,766 DEBUG awx.main.dispatch task 990f4c40-110a-4064-bada-0e948d954dff is finished
2021-04-13 11:02:58,792 DEBUG awx.main.dispatch delivered 50c4f9a9-8b6a-421d-acec-33100700f7cf to worker[180] qsize 0
2021-04-13 11:02:58,795 DEBUG awx.main.dispatch delivered 38475c6d-8b8d-453e-8caa-0c8763d7dbb5 to worker[183] qsize 0
2021-04-13 11:02:58,795 DEBUG awx.main.dispatch task 50c4f9a9-8b6a-421d-acec-33100700f7cf starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:02:58,796 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:02:58,796 DEBUG awx.main.dispatch delivered c6724419-d9c9-4080-90e5-dcf49a5ce376 to worker[182] qsize 0
2021-04-13 11:02:58,797 DEBUG awx.main.dispatch task 38475c6d-8b8d-453e-8caa-0c8763d7dbb5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:02:58,799 DEBUG awx.main.dispatch task c6724419-d9c9-4080-90e5-dcf49a5ce376 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:02:58,800 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:03:00,859 DEBUG awx.main.dispatch task 50c4f9a9-8b6a-421d-acec-33100700f7cf is finished
RESULT 2
OKREADY
2021-04-13 11:03:01,691 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:03:04,183 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-13 11:03:04,183 DEBUG awx.main.dispatch task c6724419-d9c9-4080-90e5-dcf49a5ce376 is finished
2021-04-13 11:03:06,414 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:03:06,856 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:02:48.459989+00:00
2021-04-13 11:03:11,375 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(79c33819-7982-4c30-bfd8-b57d62985586, queue=awx_private_queue)
[2021-04-13 11:03:11,375: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(79c33819-7982-4c30-bfd8-b57d62985586, queue=awx_private_queue)
2021-04-13 11:03:11,395 DEBUG awx.main.dispatch delivered 79c33819-7982-4c30-bfd8-b57d62985586 to worker[180] qsize 0
2021-04-13 11:03:11,401 DEBUG awx.main.dispatch task 79c33819-7982-4c30-bfd8-b57d62985586 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:03:11,404 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:03:11,536 DEBUG awx.main.dispatch task 38475c6d-8b8d-453e-8caa-0c8763d7dbb5 is finished
2021-04-13 11:03:20,179 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(aaa4149b-4707-489e-851d-9a38ec98efb2, queue=awx_private_queue)
[2021-04-13 11:03:20,179: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(aaa4149b-4707-489e-851d-9a38ec98efb2, queue=awx_private_queue)
2021-04-13 11:03:20,206 DEBUG awx.main.dispatch delivered aaa4149b-4707-489e-851d-9a38ec98efb2 to worker[182] qsize 0
2021-04-13 11:03:20,211 DEBUG awx.main.dispatch task aaa4149b-4707-489e-851d-9a38ec98efb2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:03:26,647 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-13 11:03:26,647 DEBUG awx.main.dispatch task 79c33819-7982-4c30-bfd8-b57d62985586 is finished
2021-04-13 11:03:26,856 DEBUG awx.main.dispatch task 996d40c7-fe74-4409-9845-fd730908e766 is finished
2021-04-13 11:03:26,945 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:03:26,999 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:03:06.414285+00:00
2021-04-13 11:03:31,373 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2c115d95-d0c7-481e-8b92-1d36ff0006bc, queue=awx_private_queue)
[2021-04-13 11:03:31,373: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2c115d95-d0c7-481e-8b92-1d36ff0006bc, queue=awx_private_queue)
2021-04-13 11:03:31,393 DEBUG awx.main.dispatch delivered 2c115d95-d0c7-481e-8b92-1d36ff0006bc to worker[180] qsize 0
2021-04-13 11:03:31,399 DEBUG awx.main.dispatch task 2c115d95-d0c7-481e-8b92-1d36ff0006bc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:03:31,400 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:03:57,049 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:03:57,114 DEBUG awx.main.dispatch task aaa4149b-4707-489e-851d-9a38ec98efb2 is finished
2021-04-13 11:03:57,755 DEBUG awx.main.dispatch task 2c115d95-d0c7-481e-8b92-1d36ff0006bc is finished
2021-04-13 11:03:49,915 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(58bdfbbe-0bc6-4b87-b89b-a2f1f966a7d2, queue=dev-awx-0)
[2021-04-13 11:03:49,915: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(58bdfbbe-0bc6-4b87-b89b-a2f1f966a7d2, queue=dev-awx-0)
2021-04-13 11:04:00,437 DEBUG awx.main.dispatch delivered 58bdfbbe-0bc6-4b87-b89b-a2f1f966a7d2 to worker[180] qsize 0
RESULT 2
OKREADY
2021-04-13 11:04:00,642 DEBUG awx.main.dispatch task 58bdfbbe-0bc6-4b87-b89b-a2f1f966a7d2 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:04:00,721 DEBUG awx.main.dispatch task 58bdfbbe-0bc6-4b87-b89b-a2f1f966a7d2 is finished
2021-04-13 11:04:00,439 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(2a74e5e9-10ba-4436-99dc-21f5c7e04855, queue=dev-awx-0)
[2021-04-13 11:04:00,439: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(2a74e5e9-10ba-4436-99dc-21f5c7e04855, queue=dev-awx-0)
2021-04-13 11:04:00,919 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(51fc8522-7ac1-4793-924f-3cfb33c9f1d3, queue=awx_private_queue)
[2021-04-13 11:04:00,919: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(51fc8522-7ac1-4793-924f-3cfb33c9f1d3, queue=awx_private_queue)
2021-04-13 11:04:00,933 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2e2bd1ee-1599-4319-bb98-abf1c5faba9b, queue=awx_private_queue)
[2021-04-13 11:04:00,933: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2e2bd1ee-1599-4319-bb98-abf1c5faba9b, queue=awx_private_queue)
2021-04-13 11:04:05,488 DEBUG awx.main.dispatch delivered 2a74e5e9-10ba-4436-99dc-21f5c7e04855 to worker[181] qsize 0
2021-04-13 11:04:05,490 DEBUG awx.main.dispatch delivered 51fc8522-7ac1-4793-924f-3cfb33c9f1d3 to worker[183] qsize 0
2021-04-13 11:04:05,490 DEBUG awx.main.dispatch task 2a74e5e9-10ba-4436-99dc-21f5c7e04855 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:04:05,493 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:04:05,496 DEBUG awx.main.dispatch delivered 2e2bd1ee-1599-4319-bb98-abf1c5faba9b to worker[182] qsize 0
2021-04-13 11:04:05,495 DEBUG awx.main.dispatch task 51fc8522-7ac1-4793-924f-3cfb33c9f1d3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:04:05,499 DEBUG awx.main.dispatch task 2e2bd1ee-1599-4319-bb98-abf1c5faba9b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:04:05,501 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:04:21,076 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6c51776d-e5c7-4321-b400-ebe9c7c29bdf, queue=awx_private_queue)
[2021-04-13 11:04:21,076: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6c51776d-e5c7-4321-b400-ebe9c7c29bdf, queue=awx_private_queue)
2021-04-13 11:04:21,094 DEBUG awx.main.dispatch delivered 6c51776d-e5c7-4321-b400-ebe9c7c29bdf to worker[180] qsize 0
2021-04-13 11:04:21,097 DEBUG awx.main.dispatch task 6c51776d-e5c7-4321-b400-ebe9c7c29bdf starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:04:21,098 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:04:24,431 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:04:27,076 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:04:30,919 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4d2ad632-f520-46e9-a2ca-36cfeeaa1f07, queue=awx_private_queue)
[2021-04-13 11:04:30,919: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4d2ad632-f520-46e9-a2ca-36cfeeaa1f07, queue=awx_private_queue)
2021-04-13 11:04:30,957 WARNING awx.main.dispatch scaling up worker pid:192
2021-04-13 11:04:30,962 DEBUG awx.main.dispatch delivered 4d2ad632-f520-46e9-a2ca-36cfeeaa1f07 to worker[192] qsize 0
2021-04-13 11:04:30,966 DEBUG awx.main.dispatch task 4d2ad632-f520-46e9-a2ca-36cfeeaa1f07 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:04:34,504 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-04-13 11:04:34,504 DEBUG awx.main.dispatch task 4d2ad632-f520-46e9-a2ca-36cfeeaa1f07 is finished
2021-04-13 11:04:34,523 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-13 11:04:34,524 DEBUG awx.main.dispatch task 6c51776d-e5c7-4321-b400-ebe9c7c29bdf is finished
2021-04-13 11:04:34,540 DEBUG awx.main.dispatch task 2e2bd1ee-1599-4319-bb98-abf1c5faba9b is finished
2021-04-13 11:04:34,647 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:03:26.946191+00:00
2021-04-13 11:04:34,711 DEBUG awx.main.dispatch task 51fc8522-7ac1-4793-924f-3cfb33c9f1d3 is finished
2021-04-13 11:04:35,324 DEBUG awx.main.dispatch task 2a74e5e9-10ba-4436-99dc-21f5c7e04855 is finished
2021-04-13 11:04:41,081 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cb794dd3-5240-4665-bcae-b55f0c4e00bd, queue=awx_private_queue)
[2021-04-13 11:04:41,081: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cb794dd3-5240-4665-bcae-b55f0c4e00bd, queue=awx_private_queue)
2021-04-13 11:04:41,101 DEBUG awx.main.dispatch delivered cb794dd3-5240-4665-bcae-b55f0c4e00bd to worker[180] qsize 0
2021-04-13 11:04:41,106 DEBUG awx.main.dispatch task cb794dd3-5240-4665-bcae-b55f0c4e00bd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:04:41,109 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:04:41,150 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:04:41,219 DEBUG awx.main.dispatch task cb794dd3-5240-4665-bcae-b55f0c4e00bd is finished
2021-04-13 11:04:49,916 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(593d47d4-9c1a-4676-a44e-9db9681bd893, queue=dev-awx-0)
[2021-04-13 11:04:49,916: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(593d47d4-9c1a-4676-a44e-9db9681bd893, queue=dev-awx-0)
2021-04-13 11:04:49,936 DEBUG awx.main.dispatch delivered 593d47d4-9c1a-4676-a44e-9db9681bd893 to worker[180] qsize 0
2021-04-13 11:04:49,940 DEBUG awx.main.dispatch task 593d47d4-9c1a-4676-a44e-9db9681bd893 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:04:50,117 DEBUG awx.main.dispatch task 593d47d4-9c1a-4676-a44e-9db9681bd893 is finished
2021-04-13 11:05:00,440 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(4d529ca3-670f-41ce-a6e7-b0a4a6888e90, queue=dev-awx-0)
[2021-04-13 11:05:00,440: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(4d529ca3-670f-41ce-a6e7-b0a4a6888e90, queue=dev-awx-0)
RESULT 2
OKREADY
2021-04-13 11:05:00,465 WARNING awx.main.dispatch scaling down worker pid:180
2021-04-13 11:05:00,469 WARNING awx.main.dispatch worker exiting gracefully pid:180
2021-04-13 11:05:00,560 DEBUG awx.main.dispatch delivered 4d529ca3-670f-41ce-a6e7-b0a4a6888e90 to worker[183] qsize 0
2021-04-13 11:05:00,565 DEBUG awx.main.dispatch task 4d529ca3-670f-41ce-a6e7-b0a4a6888e90 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:05:00,569 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:05:00,699 DEBUG awx.main.dispatch task 4d529ca3-670f-41ce-a6e7-b0a4a6888e90 is finished
2021-04-13 11:05:00,919 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6dd33d92-8824-461c-8e65-03b3ba8b0f98, queue=awx_private_queue)
[2021-04-13 11:05:00,919: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6dd33d92-8824-461c-8e65-03b3ba8b0f98, queue=awx_private_queue)
2021-04-13 11:05:00,956 DEBUG awx.main.dispatch delivered 6dd33d92-8824-461c-8e65-03b3ba8b0f98 to worker[181] qsize 0
2021-04-13 11:05:00,960 DEBUG awx.main.dispatch task 6dd33d92-8824-461c-8e65-03b3ba8b0f98 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:05:00,973 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:05:00,977 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:04:27.076503+00:00
2021-04-13 11:05:01,032 DEBUG awx.main.dispatch task 6dd33d92-8824-461c-8e65-03b3ba8b0f98 is finished
2021-04-13 11:05:01,078 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b8c35a88-bcf0-4f25-9ef9-0c249fbd040b, queue=awx_private_queue)
[2021-04-13 11:05:01,078: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b8c35a88-bcf0-4f25-9ef9-0c249fbd040b, queue=awx_private_queue)
2021-04-13 11:05:01,092 DEBUG awx.main.dispatch delivered b8c35a88-bcf0-4f25-9ef9-0c249fbd040b to worker[183] qsize 0
2021-04-13 11:05:01,095 DEBUG awx.main.dispatch task b8c35a88-bcf0-4f25-9ef9-0c249fbd040b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:05:01,096 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:05:01,143 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:05:01,210 DEBUG awx.main.dispatch task b8c35a88-bcf0-4f25-9ef9-0c249fbd040b is finished
2021-04-13 11:05:21,084 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(44bff930-5314-4412-91eb-c8ecc2f7883d, queue=awx_private_queue)
[2021-04-13 11:05:21,084: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(44bff930-5314-4412-91eb-c8ecc2f7883d, queue=awx_private_queue)
2021-04-13 11:05:21,104 DEBUG awx.main.dispatch delivered 44bff930-5314-4412-91eb-c8ecc2f7883d to worker[182] qsize 0
2021-04-13 11:05:21,109 DEBUG awx.main.dispatch task 44bff930-5314-4412-91eb-c8ecc2f7883d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:05:21,113 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:05:21,542 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:05:21,595 DEBUG awx.main.dispatch task 44bff930-5314-4412-91eb-c8ecc2f7883d is finished
2021-04-13 11:05:30,919 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f2824d85-5316-4b01-84bf-227c748bbee4, queue=awx_private_queue)
[2021-04-13 11:05:30,919: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f2824d85-5316-4b01-84bf-227c748bbee4, queue=awx_private_queue)
2021-04-13 11:05:30,934 DEBUG awx.main.dispatch delivered f2824d85-5316-4b01-84bf-227c748bbee4 to worker[181] qsize 0
2021-04-13 11:05:30,937 DEBUG awx.main.dispatch task f2824d85-5316-4b01-84bf-227c748bbee4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:05:32,287 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:05:32,293 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:05:00.973330+00:00
2021-04-13 11:05:32,328 DEBUG awx.main.dispatch task f2824d85-5316-4b01-84bf-227c748bbee4 is finished
2021-04-13 11:05:41,084 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(33447357-64de-4b74-95bc-2785409acfbf, queue=awx_private_queue)
[2021-04-13 11:05:41,084: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(33447357-64de-4b74-95bc-2785409acfbf, queue=awx_private_queue)
2021-04-13 11:05:41,103 DEBUG awx.main.dispatch delivered 33447357-64de-4b74-95bc-2785409acfbf to worker[183] qsize 0
2021-04-13 11:05:41,105 DEBUG awx.main.dispatch task 33447357-64de-4b74-95bc-2785409acfbf starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:05:42,778 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:05:43,467 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:05:43,587 DEBUG awx.main.dispatch task 33447357-64de-4b74-95bc-2785409acfbf is finished
2021-04-13 11:05:49,915 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2055afd0-3e7e-4fef-b981-7cf7f14b8b55, queue=dev-awx-0)
[2021-04-13 11:05:49,915: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2055afd0-3e7e-4fef-b981-7cf7f14b8b55, queue=dev-awx-0)
2021-04-13 11:05:49,934 DEBUG awx.main.dispatch delivered 2055afd0-3e7e-4fef-b981-7cf7f14b8b55 to worker[192] qsize 0
2021-04-13 11:05:49,939 DEBUG awx.main.dispatch task 2055afd0-3e7e-4fef-b981-7cf7f14b8b55 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:05:50,075 DEBUG awx.main.dispatch task 2055afd0-3e7e-4fef-b981-7cf7f14b8b55 is finished
2021-04-13 11:06:00,447 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6ac0f72c-789e-4b67-a523-8df9b23ffa42, queue=dev-awx-0)
[2021-04-13 11:06:00,447: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6ac0f72c-789e-4b67-a523-8df9b23ffa42, queue=dev-awx-0)
RESULT 2
OKREADY
2021-04-13 11:06:00,919 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9b5f9f28-6bd9-4000-abf3-df31e674e8f6, queue=awx_private_queue)
[2021-04-13 11:06:00,919: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9b5f9f28-6bd9-4000-abf3-df31e674e8f6, queue=awx_private_queue)
2021-04-13 11:06:01,085 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6ecbfe2b-113e-43ca-a551-dee854327c7d, queue=awx_private_queue)
[2021-04-13 11:06:01,085: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6ecbfe2b-113e-43ca-a551-dee854327c7d, queue=awx_private_queue)
2021-04-13 11:06:21,089 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(24240188-d6ea-4b62-858c-28da4b3f8eb3, queue=awx_private_queue)
[2021-04-13 11:06:21,089: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(24240188-d6ea-4b62-858c-28da4b3f8eb3, queue=awx_private_queue)
2021-04-13 11:06:28,857 DEBUG awx.main.dispatch delivered 6ac0f72c-789e-4b67-a523-8df9b23ffa42 to worker[192] qsize 0
2021-04-13 11:06:28,862 DEBUG awx.main.dispatch delivered 9b5f9f28-6bd9-4000-abf3-df31e674e8f6 to worker[182] qsize 0
2021-04-13 11:06:28,865 DEBUG awx.main.dispatch delivered 6ecbfe2b-113e-43ca-a551-dee854327c7d to worker[183] qsize 0
2021-04-13 11:06:28,868 DEBUG awx.main.dispatch delivered 24240188-d6ea-4b62-858c-28da4b3f8eb3 to worker[181] qsize 0
2021-04-13 11:06:28,863 DEBUG awx.main.dispatch task 6ac0f72c-789e-4b67-a523-8df9b23ffa42 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:06:28,870 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:06:28,868 DEBUG awx.main.dispatch task 9b5f9f28-6bd9-4000-abf3-df31e674e8f6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:06:28,873 DEBUG awx.main.dispatch task 6ecbfe2b-113e-43ca-a551-dee854327c7d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:06:28,877 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:06:28,872 DEBUG awx.main.dispatch task 24240188-d6ea-4b62-858c-28da4b3f8eb3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:06:28,878 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:06:30,919 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e28a29a9-9521-4cbb-ae94-a876560e661c, queue=awx_private_queue)
[2021-04-13 11:06:30,919: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e28a29a9-9521-4cbb-ae94-a876560e661c, queue=awx_private_queue)
2021-04-13 11:06:30,954 WARNING awx.main.dispatch scaling up worker pid:195
2021-04-13 11:06:30,959 DEBUG awx.main.dispatch delivered e28a29a9-9521-4cbb-ae94-a876560e661c to worker[195] qsize 0
2021-04-13 11:06:30,964 DEBUG awx.main.dispatch task e28a29a9-9521-4cbb-ae94-a876560e661c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:06:31,960 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:06:31,960 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-04-13 11:06:31,960 DEBUG awx.main.dispatch task e28a29a9-9521-4cbb-ae94-a876560e661c is finished
2021-04-13 11:06:36,244 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:05:32.288284+00:00
2021-04-13 11:06:37,886 DEBUG awx.main.dispatch task 6ac0f72c-789e-4b67-a523-8df9b23ffa42 is finished
2021-04-13 11:06:37,948 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:06:39,442 DEBUG awx.main.dispatch task 9b5f9f28-6bd9-4000-abf3-df31e674e8f6 is finished
2021-04-13 11:06:40,360 DEBUG awx.main.dispatch task 6ecbfe2b-113e-43ca-a551-dee854327c7d is finished
2021-04-13 11:06:41,089 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cdfe4493-230b-442a-a997-d765376bc14d, queue=awx_private_queue)
[2021-04-13 11:06:41,089: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cdfe4493-230b-442a-a997-d765376bc14d, queue=awx_private_queue)
2021-04-13 11:06:41,106 DEBUG awx.main.dispatch delivered cdfe4493-230b-442a-a997-d765376bc14d to worker[182] qsize 0
2021-04-13 11:06:41,109 DEBUG awx.main.dispatch task cdfe4493-230b-442a-a997-d765376bc14d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:06:41,110 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:06:48,767 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:06:48,813 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-13 11:06:48,814 DEBUG awx.main.dispatch task 24240188-d6ea-4b62-858c-28da4b3f8eb3 is finished
2021-04-13 11:06:48,877 DEBUG awx.main.dispatch task cdfe4493-230b-442a-a997-d765376bc14d is finished
2021-04-13 11:06:49,915 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3c9284fa-3a02-4724-b39e-18363c9f5bce, queue=dev-awx-0)
[2021-04-13 11:06:49,915: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3c9284fa-3a02-4724-b39e-18363c9f5bce, queue=dev-awx-0)
2021-04-13 11:06:49,932 DEBUG awx.main.dispatch delivered 3c9284fa-3a02-4724-b39e-18363c9f5bce to worker[181] qsize 0
2021-04-13 11:06:49,935 DEBUG awx.main.dispatch task 3c9284fa-3a02-4724-b39e-18363c9f5bce starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:06:52,669 DEBUG awx.main.dispatch task 3c9284fa-3a02-4724-b39e-18363c9f5bce is finished
2021-04-13 11:07:00,443 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5c8990ce-f15b-48ec-a1ec-763c75f5df61, queue=dev-awx-0)
[2021-04-13 11:07:00,443: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5c8990ce-f15b-48ec-a1ec-763c75f5df61, queue=dev-awx-0)
2021-04-13 11:07:00,461 WARNING awx.main.dispatch scaling down worker pid:181
RESULT 2
OKREADY
2021-04-13 11:07:00,465 WARNING awx.main.dispatch worker exiting gracefully pid:181
2021-04-13 11:07:00,919 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d3004846-8eec-4828-878f-fb3571e26aef, queue=awx_private_queue)
[2021-04-13 11:07:00,919: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d3004846-8eec-4828-878f-fb3571e26aef, queue=awx_private_queue)
2021-04-13 11:07:10,992 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c096b084-169a-4111-a98f-fd9f4fc09b2e, queue=awx_private_queue)
[2021-04-13 11:07:10,992: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c096b084-169a-4111-a98f-fd9f4fc09b2e, queue=awx_private_queue)
2021-04-13 11:07:12,979 DEBUG awx.main.dispatch delivered 5c8990ce-f15b-48ec-a1ec-763c75f5df61 to worker[192] qsize 0
2021-04-13 11:07:12,982 DEBUG awx.main.dispatch delivered d3004846-8eec-4828-878f-fb3571e26aef to worker[195] qsize 0
2021-04-13 11:07:12,985 DEBUG awx.main.dispatch delivered c096b084-169a-4111-a98f-fd9f4fc09b2e to worker[182] qsize 0
2021-04-13 11:07:12,986 DEBUG awx.main.dispatch task d3004846-8eec-4828-878f-fb3571e26aef starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:07:12,989 DEBUG awx.main.dispatch task c096b084-169a-4111-a98f-fd9f4fc09b2e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:07:12,994 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:07:12,984 DEBUG awx.main.dispatch task 5c8990ce-f15b-48ec-a1ec-763c75f5df61 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:07:12,995 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:07:13,049 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:07:13,058 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:07:13,058 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:06:31.960696+00:00
2021-04-13 11:07:13,147 DEBUG awx.main.dispatch task 5c8990ce-f15b-48ec-a1ec-763c75f5df61 is finished
2021-04-13 11:07:24,987 DEBUG awx.main.dispatch task d3004846-8eec-4828-878f-fb3571e26aef is finished
2021-04-13 11:07:25,121 DEBUG awx.main.dispatch task c096b084-169a-4111-a98f-fd9f4fc09b2e is finished
2021-04-13 11:07:31,121 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b2221385-8b03-4b22-8ea1-c7885aa4951d, queue=awx_private_queue)
[2021-04-13 11:07:31,121: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b2221385-8b03-4b22-8ea1-c7885aa4951d, queue=awx_private_queue)
2021-04-13 11:07:31,142 DEBUG awx.main.dispatch delivered b2221385-8b03-4b22-8ea1-c7885aa4951d to worker[192] qsize 0
2021-04-13 11:07:31,145 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a29da93f-195d-47a3-af0e-416da3e40dd4, queue=awx_private_queue)
[2021-04-13 11:07:31,145: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a29da93f-195d-47a3-af0e-416da3e40dd4, queue=awx_private_queue)
2021-04-13 11:07:31,149 DEBUG awx.main.dispatch task b2221385-8b03-4b22-8ea1-c7885aa4951d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:07:31,151 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:07:31,161 DEBUG awx.main.dispatch delivered a29da93f-195d-47a3-af0e-416da3e40dd4 to worker[182] qsize 0
2021-04-13 11:07:31,163 DEBUG awx.main.dispatch task a29da93f-195d-47a3-af0e-416da3e40dd4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:07:33,473 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:07:33,489 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:07:13.049396+00:00
2021-04-13 11:07:34,377 DEBUG awx.main.dispatch task a29da93f-195d-47a3-af0e-416da3e40dd4 is finished
2021-04-13 11:07:34,438 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:07:36,124 DEBUG awx.main.dispatch task b2221385-8b03-4b22-8ea1-c7885aa4951d is finished
2021-04-13 11:07:49,924 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8c2679a0-e93b-46e4-b3bd-432e3319d636, queue=dev-awx-0)
[2021-04-13 11:07:49,924: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8c2679a0-e93b-46e4-b3bd-432e3319d636, queue=dev-awx-0)
2021-04-13 11:07:49,941 DEBUG awx.main.dispatch delivered 8c2679a0-e93b-46e4-b3bd-432e3319d636 to worker[182] qsize 0
2021-04-13 11:07:49,945 DEBUG awx.main.dispatch task 8c2679a0-e93b-46e4-b3bd-432e3319d636 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:07:51,119 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c5e801ed-4291-46c4-9bc3-6d406abf224f, queue=awx_private_queue)
[2021-04-13 11:07:51,119: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c5e801ed-4291-46c4-9bc3-6d406abf224f, queue=awx_private_queue)
2021-04-13 11:07:51,135 DEBUG awx.main.dispatch delivered c5e801ed-4291-46c4-9bc3-6d406abf224f to worker[192] qsize 0
2021-04-13 11:07:51,138 DEBUG awx.main.dispatch task c5e801ed-4291-46c4-9bc3-6d406abf224f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:07:51,140 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:07:53,427 DEBUG awx.main.dispatch task 8c2679a0-e93b-46e4-b3bd-432e3319d636 is finished
2021-04-13 11:07:53,430 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:07:53,469 DEBUG awx.main.dispatch task c5e801ed-4291-46c4-9bc3-6d406abf224f is finished
2021-04-13 11:08:00,444 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(ba6a8369-9302-49b7-8632-3961bc77c6e7, queue=dev-awx-0)
[2021-04-13 11:08:00,444: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(ba6a8369-9302-49b7-8632-3961bc77c6e7, queue=dev-awx-0)
2021-04-13 11:08:01,145 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(924dd575-1b30-455f-87f7-e19b81ac63b6, queue=awx_private_queue)
[2021-04-13 11:08:01,145: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(924dd575-1b30-455f-87f7-e19b81ac63b6, queue=awx_private_queue)
RESULT 2
OKREADY
2021-04-13 11:08:01,852 DEBUG awx.main.dispatch delivered ba6a8369-9302-49b7-8632-3961bc77c6e7 to worker[182] qsize 0
2021-04-13 11:08:01,854 DEBUG awx.main.dispatch task ba6a8369-9302-49b7-8632-3961bc77c6e7 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:08:01,856 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:08:01,856 DEBUG awx.main.dispatch delivered 924dd575-1b30-455f-87f7-e19b81ac63b6 to worker[192] qsize 0
2021-04-13 11:08:01,859 DEBUG awx.main.dispatch task 924dd575-1b30-455f-87f7-e19b81ac63b6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:08:06,527 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:08:06,681 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:07:33.474286+00:00
2021-04-13 11:08:06,699 DEBUG awx.main.dispatch task ba6a8369-9302-49b7-8632-3961bc77c6e7 is finished
2021-04-13 11:08:06,721 DEBUG awx.main.dispatch task 924dd575-1b30-455f-87f7-e19b81ac63b6 is finished
2021-04-13 11:08:11,119 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9b68bd83-9f59-4616-b163-9e8f68c35bcd, queue=awx_private_queue)
[2021-04-13 11:08:11,119: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9b68bd83-9f59-4616-b163-9e8f68c35bcd, queue=awx_private_queue)
2021-04-13 11:08:13,779 DEBUG awx.main.dispatch delivered 9b68bd83-9f59-4616-b163-9e8f68c35bcd to worker[182] qsize 0
2021-04-13 11:08:13,782 DEBUG awx.main.dispatch task 9b68bd83-9f59-4616-b163-9e8f68c35bcd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:08:13,784 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:08:15,091 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:08:15,191 DEBUG awx.main.dispatch task 9b68bd83-9f59-4616-b163-9e8f68c35bcd is finished
2021-04-13 11:08:31,148 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(813426ca-cb5a-4bf8-b35d-f20cd26cbcb3, queue=awx_private_queue)
[2021-04-13 11:08:31,148: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(813426ca-cb5a-4bf8-b35d-f20cd26cbcb3, queue=awx_private_queue)
2021-04-13 11:08:31,168 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cbf41584-f24b-4351-92dc-21e73d485356, queue=awx_private_queue)
[2021-04-13 11:08:31,168: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cbf41584-f24b-4351-92dc-21e73d485356, queue=awx_private_queue)
2021-04-13 11:08:31,167 DEBUG awx.main.dispatch delivered 813426ca-cb5a-4bf8-b35d-f20cd26cbcb3 to worker[183] qsize 0
2021-04-13 11:08:33,024 DEBUG awx.main.dispatch delivered cbf41584-f24b-4351-92dc-21e73d485356 to worker[182] qsize 0
2021-04-13 11:08:33,027 DEBUG awx.main.dispatch task cbf41584-f24b-4351-92dc-21e73d485356 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:08:33,025 DEBUG awx.main.dispatch task 813426ca-cb5a-4bf8-b35d-f20cd26cbcb3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:08:33,031 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:08:36,861 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:08:36,871 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:08:06.527749+00:00
2021-04-13 11:08:36,896 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:08:36,932 DEBUG awx.main.dispatch task 813426ca-cb5a-4bf8-b35d-f20cd26cbcb3 is finished
2021-04-13 11:08:36,963 DEBUG awx.main.dispatch task cbf41584-f24b-4351-92dc-21e73d485356 is finished
2021-04-13 11:08:49,935 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(693a6afe-3d4f-4ae2-9ad8-1bdfa3033720, queue=dev-awx-0)
[2021-04-13 11:08:49,935: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(693a6afe-3d4f-4ae2-9ad8-1bdfa3033720, queue=dev-awx-0)
2021-04-13 11:08:49,954 DEBUG awx.main.dispatch delivered 693a6afe-3d4f-4ae2-9ad8-1bdfa3033720 to worker[195] qsize 0
2021-04-13 11:08:49,958 DEBUG awx.main.dispatch task 693a6afe-3d4f-4ae2-9ad8-1bdfa3033720 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:08:50,023 DEBUG awx.main.dispatch task 693a6afe-3d4f-4ae2-9ad8-1bdfa3033720 is finished
2021-04-13 11:08:51,168 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0af782f2-b7dd-4e51-9fa1-9733c99e6eaa, queue=awx_private_queue)
[2021-04-13 11:08:51,168: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0af782f2-b7dd-4e51-9fa1-9733c99e6eaa, queue=awx_private_queue)
2021-04-13 11:08:51,183 DEBUG awx.main.dispatch delivered 0af782f2-b7dd-4e51-9fa1-9733c99e6eaa to worker[195] qsize 0
2021-04-13 11:08:51,187 DEBUG awx.main.dispatch task 0af782f2-b7dd-4e51-9fa1-9733c99e6eaa starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:08:51,188 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:08:51,263 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:08:51,546 DEBUG awx.main.dispatch task 0af782f2-b7dd-4e51-9fa1-9733c99e6eaa is finished
2021-04-13 11:09:00,444 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6621e48c-3d9b-45b7-8608-1b363636c9db, queue=dev-awx-0)
[2021-04-13 11:09:00,444: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6621e48c-3d9b-45b7-8608-1b363636c9db, queue=dev-awx-0)
2021-04-13 11:09:01,145 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(be983fab-712a-42d3-a3aa-68d6d4f2b8a1, queue=awx_private_queue)
[2021-04-13 11:09:01,145: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(be983fab-712a-42d3-a3aa-68d6d4f2b8a1, queue=awx_private_queue)
RESULT 2
OKREADY
2021-04-13 11:09:02,140 DEBUG awx.main.dispatch delivered 6621e48c-3d9b-45b7-8608-1b363636c9db to worker[192] qsize 0
2021-04-13 11:09:02,142 DEBUG awx.main.dispatch delivered be983fab-712a-42d3-a3aa-68d6d4f2b8a1 to worker[183] qsize 0
2021-04-13 11:09:02,144 DEBUG awx.main.dispatch task be983fab-712a-42d3-a3aa-68d6d4f2b8a1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:09:02,144 DEBUG awx.main.dispatch task 6621e48c-3d9b-45b7-8608-1b363636c9db starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:09:02,148 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:09:02,224 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:09:02,232 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:08:36.863336+00:00
2021-04-13 11:09:02,701 DEBUG awx.main.dispatch task be983fab-712a-42d3-a3aa-68d6d4f2b8a1 is finished
2021-04-13 11:09:02,778 DEBUG awx.main.dispatch task 6621e48c-3d9b-45b7-8608-1b363636c9db is finished
2021-04-13 11:09:11,169 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b060952e-e695-4417-b2fa-5d11fd12d07b, queue=awx_private_queue)
[2021-04-13 11:09:11,169: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b060952e-e695-4417-b2fa-5d11fd12d07b, queue=awx_private_queue)
2021-04-13 11:09:11,186 DEBUG awx.main.dispatch delivered b060952e-e695-4417-b2fa-5d11fd12d07b to worker[183] qsize 0
2021-04-13 11:09:11,190 DEBUG awx.main.dispatch task b060952e-e695-4417-b2fa-5d11fd12d07b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:09:11,192 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:09:14,466 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:09:14,547 DEBUG awx.main.dispatch task b060952e-e695-4417-b2fa-5d11fd12d07b is finished
2021-04-13 11:09:31,156 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ae89c603-1c8a-496f-9dd9-31581de54fdf, queue=awx_private_queue)
[2021-04-13 11:09:31,156: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ae89c603-1c8a-496f-9dd9-31581de54fdf, queue=awx_private_queue)
2021-04-13 11:09:31,177 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(85fde55e-c336-4e56-9d03-3f56c62bcc68, queue=awx_private_queue)
[2021-04-13 11:09:31,177: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(85fde55e-c336-4e56-9d03-3f56c62bcc68, queue=awx_private_queue)
2021-04-13 11:09:31,176 DEBUG awx.main.dispatch delivered ae89c603-1c8a-496f-9dd9-31581de54fdf to worker[192] qsize 0
2021-04-13 11:09:31,181 DEBUG awx.main.dispatch task ae89c603-1c8a-496f-9dd9-31581de54fdf starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:09:31,193 DEBUG awx.main.dispatch delivered 85fde55e-c336-4e56-9d03-3f56c62bcc68 to worker[195] qsize 0
2021-04-13 11:09:31,196 DEBUG awx.main.dispatch task 85fde55e-c336-4e56-9d03-3f56c62bcc68 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:09:31,200 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:09:31,674 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:09:31,685 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:09:02.224271+00:00
2021-04-13 11:09:31,724 DEBUG awx.main.dispatch task ae89c603-1c8a-496f-9dd9-31581de54fdf is finished
2021-04-13 11:09:31,774 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:09:31,814 DEBUG awx.main.dispatch task 85fde55e-c336-4e56-9d03-3f56c62bcc68 is finished
2021-04-13 11:09:49,945 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(efd6ab21-2b02-4385-ae35-87ebf03519ef, queue=dev-awx-0)
[2021-04-13 11:09:49,945: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(efd6ab21-2b02-4385-ae35-87ebf03519ef, queue=dev-awx-0)
2021-04-13 11:09:55,732 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f374f8b0-67c5-4c11-ae83-2f598b6369c4, queue=awx_private_queue)
[2021-04-13 11:09:55,732: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f374f8b0-67c5-4c11-ae83-2f598b6369c4, queue=awx_private_queue)
2021-04-13 11:09:55,745 DEBUG awx.main.dispatch delivered efd6ab21-2b02-4385-ae35-87ebf03519ef to worker[183] qsize 0
2021-04-13 11:09:55,751 DEBUG awx.main.dispatch delivered f374f8b0-67c5-4c11-ae83-2f598b6369c4 to worker[182] qsize 0
2021-04-13 11:09:55,755 DEBUG awx.main.dispatch task f374f8b0-67c5-4c11-ae83-2f598b6369c4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:09:57,911 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:09:55,750 DEBUG awx.main.dispatch task efd6ab21-2b02-4385-ae35-87ebf03519ef starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:09:58,179 DEBUG awx.main.dispatch task efd6ab21-2b02-4385-ae35-87ebf03519ef is finished
2021-04-13 11:09:58,357 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:09:58,885 DEBUG awx.main.dispatch task f374f8b0-67c5-4c11-ae83-2f598b6369c4 is finished
2021-04-13 11:10:00,444 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(bf17d7fa-c081-4e29-bd74-e65771842a9d, queue=dev-awx-0)
[2021-04-13 11:10:00,444: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(bf17d7fa-c081-4e29-bd74-e65771842a9d, queue=dev-awx-0)
2021-04-13 11:10:01,155 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9420b612-c38f-4d3a-8255-b9e7a4cff415, queue=awx_private_queue)
[2021-04-13 11:10:01,155: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9420b612-c38f-4d3a-8255-b9e7a4cff415, queue=awx_private_queue)
RESULT 2
OKREADY
2021-04-13 11:10:01,674 DEBUG awx.main.dispatch delivered bf17d7fa-c081-4e29-bd74-e65771842a9d to worker[183] qsize 0
2021-04-13 11:10:01,677 DEBUG awx.main.dispatch delivered 9420b612-c38f-4d3a-8255-b9e7a4cff415 to worker[182] qsize 0
2021-04-13 11:10:01,679 DEBUG awx.main.dispatch task 9420b612-c38f-4d3a-8255-b9e7a4cff415 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:10:01,677 DEBUG awx.main.dispatch task bf17d7fa-c081-4e29-bd74-e65771842a9d starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:10:01,682 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:10:04,186 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:10:04,193 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:09:31.675219+00:00
2021-04-13 11:10:04,391 DEBUG awx.main.dispatch task bf17d7fa-c081-4e29-bd74-e65771842a9d is finished
2021-04-13 11:10:04,915 DEBUG awx.main.dispatch task 9420b612-c38f-4d3a-8255-b9e7a4cff415 is finished
2021-04-13 11:10:15,736 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f8cc6456-b925-49cd-aba0-b6d3b5df5742, queue=awx_private_queue)
[2021-04-13 11:10:15,736: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f8cc6456-b925-49cd-aba0-b6d3b5df5742, queue=awx_private_queue)
2021-04-13 11:10:15,753 DEBUG awx.main.dispatch delivered f8cc6456-b925-49cd-aba0-b6d3b5df5742 to worker[195] qsize 0
2021-04-13 11:10:15,756 DEBUG awx.main.dispatch task f8cc6456-b925-49cd-aba0-b6d3b5df5742 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:10:15,759 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:10:19,541 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:10:19,798 DEBUG awx.main.dispatch task f8cc6456-b925-49cd-aba0-b6d3b5df5742 is finished
2021-04-13 11:10:31,229 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dbea48e9-b9b4-4d0b-892f-b09ff9cd7c5a, queue=awx_private_queue)
[2021-04-13 11:10:31,229: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dbea48e9-b9b4-4d0b-892f-b09ff9cd7c5a, queue=awx_private_queue)
2021-04-13 11:10:31,247 DEBUG awx.main.dispatch delivered dbea48e9-b9b4-4d0b-892f-b09ff9cd7c5a to worker[182] qsize 0
2021-04-13 11:10:31,252 DEBUG awx.main.dispatch task dbea48e9-b9b4-4d0b-892f-b09ff9cd7c5a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:10:31,317 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:10:31,321 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:10:04.186473+00:00
2021-04-13 11:10:31,373 DEBUG awx.main.dispatch task dbea48e9-b9b4-4d0b-892f-b09ff9cd7c5a is finished
2021-04-13 11:10:35,736 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3296a7eb-5d78-4d1d-b75f-85462178ea47, queue=awx_private_queue)
[2021-04-13 11:10:35,736: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3296a7eb-5d78-4d1d-b75f-85462178ea47, queue=awx_private_queue)
2021-04-13 11:10:35,753 DEBUG awx.main.dispatch delivered 3296a7eb-5d78-4d1d-b75f-85462178ea47 to worker[183] qsize 0
2021-04-13 11:10:35,756 DEBUG awx.main.dispatch task 3296a7eb-5d78-4d1d-b75f-85462178ea47 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:10:35,758 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:10:36,200 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:10:36,255 DEBUG awx.main.dispatch task 3296a7eb-5d78-4d1d-b75f-85462178ea47 is finished
2021-04-13 11:10:49,944 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b754abf2-b128-49f0-a43f-3b31c83e6d68, queue=dev-awx-0)
[2021-04-13 11:10:49,944: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b754abf2-b128-49f0-a43f-3b31c83e6d68, queue=dev-awx-0)
2021-04-13 11:10:49,958 DEBUG awx.main.dispatch delivered b754abf2-b128-49f0-a43f-3b31c83e6d68 to worker[195] qsize 0
2021-04-13 11:10:49,962 DEBUG awx.main.dispatch task b754abf2-b128-49f0-a43f-3b31c83e6d68 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:10:51,073 DEBUG awx.main.dispatch task b754abf2-b128-49f0-a43f-3b31c83e6d68 is finished
2021-04-13 11:10:55,737 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3dc690d5-ff82-4220-902a-97a6974ef395, queue=awx_private_queue)
[2021-04-13 11:10:55,737: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3dc690d5-ff82-4220-902a-97a6974ef395, queue=awx_private_queue)
2021-04-13 11:10:55,752 DEBUG awx.main.dispatch delivered 3dc690d5-ff82-4220-902a-97a6974ef395 to worker[183] qsize 0
2021-04-13 11:10:55,754 DEBUG awx.main.dispatch task 3dc690d5-ff82-4220-902a-97a6974ef395 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:10:55,756 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:10:56,119 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:10:56,199 DEBUG awx.main.dispatch task 3dc690d5-ff82-4220-902a-97a6974ef395 is finished
RESULT 2
OKREADY
2021-04-13 11:11:00,445 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(424bea94-b2b0-486d-bb21-9e80267451a9, queue=dev-awx-0)
[2021-04-13 11:11:00,445: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(424bea94-b2b0-486d-bb21-9e80267451a9, queue=dev-awx-0)
2021-04-13 11:11:01,585 DEBUG awx.main.dispatch delivered 424bea94-b2b0-486d-bb21-9e80267451a9 to worker[195] qsize 0
2021-04-13 11:11:01,587 DEBUG awx.main.dispatch task 424bea94-b2b0-486d-bb21-9e80267451a9 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:11:01,589 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:11:01,572 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8817caeb-3fb2-447f-afa9-51a72871d744, queue=awx_private_queue)
[2021-04-13 11:11:01,572: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8817caeb-3fb2-447f-afa9-51a72871d744, queue=awx_private_queue)
2021-04-13 11:11:01,808 DEBUG awx.main.dispatch delivered 8817caeb-3fb2-447f-afa9-51a72871d744 to worker[182] qsize 0
2021-04-13 11:11:01,811 DEBUG awx.main.dispatch task 8817caeb-3fb2-447f-afa9-51a72871d744 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:11:02,081 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:11:02,093 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:10:31.317534+00:00
2021-04-13 11:11:02,120 DEBUG awx.main.dispatch task 424bea94-b2b0-486d-bb21-9e80267451a9 is finished
2021-04-13 11:11:02,121 DEBUG awx.main.dispatch task 8817caeb-3fb2-447f-afa9-51a72871d744 is finished
2021-04-13 11:11:15,744 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(435ce509-daf0-4908-9631-4c8199072276, queue=awx_private_queue)
[2021-04-13 11:11:15,744: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(435ce509-daf0-4908-9631-4c8199072276, queue=awx_private_queue)
2021-04-13 11:11:15,765 DEBUG awx.main.dispatch delivered 435ce509-daf0-4908-9631-4c8199072276 to worker[183] qsize 0
2021-04-13 11:11:15,769 DEBUG awx.main.dispatch task 435ce509-daf0-4908-9631-4c8199072276 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:11:15,792 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:11:15,955 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:11:16,082 DEBUG awx.main.dispatch task 435ce509-daf0-4908-9631-4c8199072276 is finished
2021-04-13 11:11:31,575 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e0108dd3-34af-495a-b66a-a8eeb6161a69, queue=awx_private_queue)
[2021-04-13 11:11:31,575: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e0108dd3-34af-495a-b66a-a8eeb6161a69, queue=awx_private_queue)
2021-04-13 11:11:31,590 DEBUG awx.main.dispatch delivered e0108dd3-34af-495a-b66a-a8eeb6161a69 to worker[183] qsize 0
2021-04-13 11:11:31,593 DEBUG awx.main.dispatch task e0108dd3-34af-495a-b66a-a8eeb6161a69 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:11:34,609 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:11:35,140 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:11:02.081681+00:00
2021-04-13 11:11:35,258 DEBUG awx.main.dispatch task e0108dd3-34af-495a-b66a-a8eeb6161a69 is finished
2021-04-13 11:11:35,741 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e3e98faf-c2f9-46f7-841b-65284b40cc7d, queue=awx_private_queue)
[2021-04-13 11:11:35,741: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e3e98faf-c2f9-46f7-841b-65284b40cc7d, queue=awx_private_queue)
2021-04-13 11:11:35,763 DEBUG awx.main.dispatch delivered e3e98faf-c2f9-46f7-841b-65284b40cc7d to worker[192] qsize 0
2021-04-13 11:11:35,767 DEBUG awx.main.dispatch task e3e98faf-c2f9-46f7-841b-65284b40cc7d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:11:35,772 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:11:36,124 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:11:36,680 DEBUG awx.main.dispatch task e3e98faf-c2f9-46f7-841b-65284b40cc7d is finished
2021-04-13 11:11:49,839 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(77d760b6-ab5a-4413-abe0-9990acddb905, queue=dev-awx-0)
[2021-04-13 11:11:49,839: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(77d760b6-ab5a-4413-abe0-9990acddb905, queue=dev-awx-0)
2021-04-13 11:11:49,856 DEBUG awx.main.dispatch delivered 77d760b6-ab5a-4413-abe0-9990acddb905 to worker[192] qsize 0
2021-04-13 11:11:49,861 DEBUG awx.main.dispatch task 77d760b6-ab5a-4413-abe0-9990acddb905 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-04-13 11:11:49,863 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-04-13 11:11:49,944 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(fa412fc5-2f6f-46c9-94ae-355cdad76dd3, queue=dev-awx-0)
[2021-04-13 11:11:49,944: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(fa412fc5-2f6f-46c9-94ae-355cdad76dd3, queue=dev-awx-0)
2021-04-13 11:11:49,968 DEBUG awx.main.dispatch delivered fa412fc5-2f6f-46c9-94ae-355cdad76dd3 to worker[183] qsize 0
2021-04-13 11:11:49,970 DEBUG awx.main.dispatch task fa412fc5-2f6f-46c9-94ae-355cdad76dd3 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:11:50,640 DEBUG awx.main.dispatch task fa412fc5-2f6f-46c9-94ae-355cdad76dd3 is finished
2021-04-13 11:11:55,741 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1891e851-c28b-4d78-9080-4ff49f82d657, queue=awx_private_queue)
[2021-04-13 11:11:55,741: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1891e851-c28b-4d78-9080-4ff49f82d657, queue=awx_private_queue)
2021-04-13 11:11:55,758 DEBUG awx.main.dispatch delivered 1891e851-c28b-4d78-9080-4ff49f82d657 to worker[183] qsize 0
2021-04-13 11:11:55,760 DEBUG awx.main.dispatch task 1891e851-c28b-4d78-9080-4ff49f82d657 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:11:55,762 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:11:57,995 DEBUG awx.main.dispatch task 77d760b6-ab5a-4413-abe0-9990acddb905 is finished
RESULT 2
OKREADY
2021-04-13 11:12:00,444 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5dd134dd-0a0f-472a-a0bd-cb3bfb8c8d9c, queue=dev-awx-0)
[2021-04-13 11:12:00,444: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5dd134dd-0a0f-472a-a0bd-cb3bfb8c8d9c, queue=dev-awx-0)
2021-04-13 11:12:16,483 DEBUG awx.main.dispatch delivered 5dd134dd-0a0f-472a-a0bd-cb3bfb8c8d9c to worker[182] qsize 0
2021-04-13 11:12:01,574 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2ed12f3a-804f-4868-8270-45b89a58c27f, queue=awx_private_queue)
[2021-04-13 11:12:01,574: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2ed12f3a-804f-4868-8270-45b89a58c27f, queue=awx_private_queue)
2021-04-13 11:12:17,306 DEBUG awx.main.dispatch delivered 2ed12f3a-804f-4868-8270-45b89a58c27f to worker[192] qsize 0
2021-04-13 11:12:17,307 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(805b08e6-25a0-4e64-bc7a-5375ffd4f2ba, queue=awx_private_queue)
[2021-04-13 11:12:17,307: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(805b08e6-25a0-4e64-bc7a-5375ffd4f2ba, queue=awx_private_queue)
2021-04-13 11:12:17,308 DEBUG awx.main.dispatch task 2ed12f3a-804f-4868-8270-45b89a58c27f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:12:17,320 DEBUG awx.main.dispatch delivered 805b08e6-25a0-4e64-bc7a-5375ffd4f2ba to worker[195] qsize 0
2021-04-13 11:12:17,324 DEBUG awx.main.dispatch task 805b08e6-25a0-4e64-bc7a-5375ffd4f2ba starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:12:17,329 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:12:22,357 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:12:22,496 DEBUG awx.main.dispatch task 1891e851-c28b-4d78-9080-4ff49f82d657 is finished
2021-04-13 11:12:28,213 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:12:28,992 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:12:29,173 DEBUG awx.main.dispatch task 805b08e6-25a0-4e64-bc7a-5375ffd4f2ba is finished
2021-04-13 11:12:16,493 DEBUG awx.main.dispatch task 5dd134dd-0a0f-472a-a0bd-cb3bfb8c8d9c starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:12:29,309 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:12:29,491 DEBUG awx.main.dispatch task 5dd134dd-0a0f-472a-a0bd-cb3bfb8c8d9c is finished
2021-04-13 11:12:29,737 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:11:34.609942+00:00
2021-04-13 11:12:37,321 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(10480437-cad1-4937-82b0-0767ba93258a, queue=awx_private_queue)
[2021-04-13 11:12:37,321: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(10480437-cad1-4937-82b0-0767ba93258a, queue=awx_private_queue)
2021-04-13 11:12:37,341 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(cfcf1cc7-6879-4851-847c-93941c6f61a3, queue=awx_private_queue)
[2021-04-13 11:12:37,341: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(cfcf1cc7-6879-4851-847c-93941c6f61a3, queue=awx_private_queue)
2021-04-13 11:12:37,352 DEBUG awx.main.dispatch delivered 10480437-cad1-4937-82b0-0767ba93258a to worker[182] qsize 0
2021-04-13 11:12:37,359 DEBUG awx.main.dispatch task 10480437-cad1-4937-82b0-0767ba93258a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:12:37,361 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:12:37,361 DEBUG awx.main.dispatch delivered cfcf1cc7-6879-4851-847c-93941c6f61a3 to worker[183] qsize 0
2021-04-13 11:12:37,363 DEBUG awx.main.dispatch task cfcf1cc7-6879-4851-847c-93941c6f61a3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:12:49,944 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d6e6fffc-7da3-4d81-9ba9-acc31aa695d4, queue=dev-awx-0)
[2021-04-13 11:12:49,944: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d6e6fffc-7da3-4d81-9ba9-acc31aa695d4, queue=dev-awx-0)
2021-04-13 11:12:49,960 DEBUG awx.main.dispatch delivered d6e6fffc-7da3-4d81-9ba9-acc31aa695d4 to worker[195] qsize 0
2021-04-13 11:12:49,963 DEBUG awx.main.dispatch task d6e6fffc-7da3-4d81-9ba9-acc31aa695d4 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:12:53,558 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:12:53,585 DEBUG awx.main.dispatch task d6e6fffc-7da3-4d81-9ba9-acc31aa695d4 is finished
2021-04-13 11:12:53,672 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-04-13 11:12:53,672 DEBUG awx.main.dispatch task cfcf1cc7-6879-4851-847c-93941c6f61a3 is finished
2021-04-13 11:12:53,847 DEBUG awx.main.dispatch task 10480437-cad1-4937-82b0-0767ba93258a is finished
2021-04-13 11:12:57,317 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ed2ba397-24fc-4c16-8018-21be25cad14b, queue=awx_private_queue)
[2021-04-13 11:12:57,317: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ed2ba397-24fc-4c16-8018-21be25cad14b, queue=awx_private_queue)
2021-04-13 11:12:57,333 DEBUG awx.main.dispatch delivered ed2ba397-24fc-4c16-8018-21be25cad14b to worker[182] qsize 0
2021-04-13 11:12:57,336 DEBUG awx.main.dispatch task ed2ba397-24fc-4c16-8018-21be25cad14b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:12:57,337 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:13:00,444 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(be4d522b-6861-42db-936a-ee10ad7609c3, queue=dev-awx-0)
[2021-04-13 11:13:00,444: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(be4d522b-6861-42db-936a-ee10ad7609c3, queue=dev-awx-0)
RESULT 2
OKREADY
2021-04-13 11:13:07,342 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(80ed8c64-5e1d-4d9f-9e80-121418ad4e3d, queue=awx_private_queue)
[2021-04-13 11:13:07,342: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(80ed8c64-5e1d-4d9f-9e80-121418ad4e3d, queue=awx_private_queue)
2021-04-13 11:13:17,318 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(81e3595e-947b-4fb5-bb75-d5ef554b3370, queue=awx_private_queue)
[2021-04-13 11:13:17,318: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(81e3595e-947b-4fb5-bb75-d5ef554b3370, queue=awx_private_queue)
2021-04-13 11:13:38,036 DEBUG awx.main.dispatch task 2ed12f3a-804f-4868-8270-45b89a58c27f is finished
2021-04-13 11:13:40,157 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:13:44,881 DEBUG awx.main.dispatch delivered be4d522b-6861-42db-936a-ee10ad7609c3 to worker[192] qsize 0
2021-04-13 11:13:44,887 DEBUG awx.main.dispatch delivered 80ed8c64-5e1d-4d9f-9e80-121418ad4e3d to worker[195] qsize 0
2021-04-13 11:13:44,889 DEBUG awx.main.dispatch delivered 81e3595e-947b-4fb5-bb75-d5ef554b3370 to worker[183] qsize 0
2021-04-13 11:13:44,887 DEBUG awx.main.dispatch task be4d522b-6861-42db-936a-ee10ad7609c3 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:13:44,891 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:13:44,892 DEBUG awx.main.dispatch task 81e3595e-947b-4fb5-bb75-d5ef554b3370 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:13:44,897 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:13:44,891 DEBUG awx.main.dispatch task 80ed8c64-5e1d-4d9f-9e80-121418ad4e3d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:13:37,382 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(919f78d8-ecab-4638-a4d5-dbedf2fe0c11, queue=awx_private_queue)
[2021-04-13 11:13:37,382: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(919f78d8-ecab-4638-a4d5-dbedf2fe0c11, queue=awx_private_queue)
2021-04-13 11:13:51,685 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ddd66af1-823b-4918-8f43-d61d3f462f8a, queue=awx_private_queue)
[2021-04-13 11:13:51,685: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ddd66af1-823b-4918-8f43-d61d3f462f8a, queue=awx_private_queue)
2021-04-13 11:13:51,706 WARNING awx.main.dispatch scaling up worker pid:198
2021-04-13 11:13:51,715 DEBUG awx.main.dispatch delivered 919f78d8-ecab-4638-a4d5-dbedf2fe0c11 to worker[198] qsize 0
2021-04-13 11:13:51,723 DEBUG awx.main.dispatch task 919f78d8-ecab-4638-a4d5-dbedf2fe0c11 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:13:51,726 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:13:51,726 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5eee89b3-141e-49cd-bb97-eada929ace3a, queue=dev-awx-0)
[2021-04-13 11:13:51,726: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5eee89b3-141e-49cd-bb97-eada929ace3a, queue=dev-awx-0)
2021-04-13 11:13:51,733 WARNING awx.main.dispatch scaling up worker pid:200
2021-04-13 11:13:51,737 DEBUG awx.main.dispatch delivered ddd66af1-823b-4918-8f43-d61d3f462f8a to worker[200] qsize 0
2021-04-13 11:13:51,741 DEBUG awx.main.dispatch task ddd66af1-823b-4918-8f43-d61d3f462f8a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:13:51,748 WARNING awx.main.dispatch scaling up worker pid:202
2021-04-13 11:13:51,752 DEBUG awx.main.dispatch delivered 5eee89b3-141e-49cd-bb97-eada929ace3a to worker[202] qsize 0
2021-04-13 11:13:51,756 DEBUG awx.main.dispatch task 5eee89b3-141e-49cd-bb97-eada929ace3a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:13:54,076 DEBUG awx.main.dispatch task ed2ba397-24fc-4c16-8018-21be25cad14b is finished
2021-04-13 11:14:00,444 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(75b10717-4ccb-4612-8546-50b7c5ed2c9b, queue=dev-awx-0)
[2021-04-13 11:14:00,444: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(75b10717-4ccb-4612-8546-50b7c5ed2c9b, queue=dev-awx-0)
RESULT 2
OKREADY
2021-04-13 11:14:00,463 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(842f3c00-6ec1-417f-9234-b0e9c9aaba49, queue=awx_private_queue)
[2021-04-13 11:14:00,463: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(842f3c00-6ec1-417f-9234-b0e9c9aaba49, queue=awx_private_queue)
2021-04-13 11:14:00,463 WARNING awx.main.dispatch scaling down worker pid:182
2021-04-13 11:14:00,466 WARNING awx.main.dispatch worker exiting gracefully pid:182
2021-04-13 11:14:10,413 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:14:20,472 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7dab6cc1-7e90-42ab-99a5-b7880cd9b575, queue=awx_private_queue)
[2021-04-13 11:14:20,472: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7dab6cc1-7e90-42ab-99a5-b7880cd9b575, queue=awx_private_queue)
2021-04-13 11:14:21,684 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8e3f2b20-9206-45fa-9e37-42a467d3b8b1, queue=awx_private_queue)
[2021-04-13 11:14:21,684: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8e3f2b20-9206-45fa-9e37-42a467d3b8b1, queue=awx_private_queue)
2021-04-13 11:14:25,115 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:12:28.213813+00:00
2021-04-13 11:14:25,341 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:14:40,479 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(efa5c8d8-f11c-421f-8034-ea9646d3c58b, queue=awx_private_queue)
[2021-04-13 11:14:40,479: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(efa5c8d8-f11c-421f-8034-ea9646d3c58b, queue=awx_private_queue)
RESULT 2
OKREADY
2021-04-13 11:15:37,639 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-13 11:15:37,639 DEBUG awx.main.dispatch task 81e3595e-947b-4fb5-bb75-d5ef554b3370 is finished
2021-04-13 11:14:51,686 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(fadb7c9d-6b09-49d6-a11f-d43d84cb7323, queue=awx_private_queue)
[2021-04-13 11:14:51,686: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(fadb7c9d-6b09-49d6-a11f-d43d84cb7323, queue=awx_private_queue)
2021-04-13 11:15:49,103 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f397950f-bf60-4ec8-9621-84971c663c35, queue=dev-awx-0)
[2021-04-13 11:15:49,103: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f397950f-bf60-4ec8-9621-84971c663c35, queue=dev-awx-0)
2021-04-13 11:15:49,121 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(1dfa5967-e522-4017-850b-dc01eee9836c, queue=dev-awx-0)
[2021-04-13 11:15:49,121: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(1dfa5967-e522-4017-850b-dc01eee9836c, queue=dev-awx-0)
2021-04-13 11:15:49,137 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d0d6bed8-2e87-4a8f-b692-dbe4c5a60c67, queue=awx_private_queue)
[2021-04-13 11:15:49,137: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d0d6bed8-2e87-4a8f-b692-dbe4c5a60c67, queue=awx_private_queue)
RESULT 2
OKREADY
2021-04-13 11:16:09,138 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(91fbc748-2f7e-445c-8cd0-c625384fa85f, queue=awx_private_queue)
[2021-04-13 11:16:09,138: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(91fbc748-2f7e-445c-8cd0-c625384fa85f, queue=awx_private_queue)
2021-04-13 11:16:09,157 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e59122f5-c93a-4c1b-b6a9-9adab04e008a, queue=awx_private_queue)
[2021-04-13 11:16:09,157: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e59122f5-c93a-4c1b-b6a9-9adab04e008a, queue=awx_private_queue)
2021-04-13 11:16:27,129 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:16:29,145 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bf221498-3b81-4a08-a31a-039a3d940ed3, queue=awx_private_queue)
[2021-04-13 11:16:29,145: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bf221498-3b81-4a08-a31a-039a3d940ed3, queue=awx_private_queue)
2021-04-13 11:16:39,272 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6f5f8c03-36a2-42e1-81c7-9793351473af, queue=awx_private_queue)
[2021-04-13 11:16:39,272: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6f5f8c03-36a2-42e1-81c7-9793351473af, queue=awx_private_queue)
2021-04-13 11:16:53,422 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:14:10.413766+00:00
2021-04-13 11:15:43,236 DEBUG awx.main.dispatch task 80ed8c64-5e1d-4d9f-9e80-121418ad4e3d is finished
2021-04-13 11:16:56,396 DEBUG awx.main.dispatch task 5eee89b3-141e-49cd-bb97-eada929ace3a is finished
RESULT 2
OKREADY
2021-04-13 11:17:30,533 DEBUG awx.main.dispatch task be4d522b-6861-42db-936a-ee10ad7609c3 is finished
RESULT 2
OKREADY
2021-04-13 11:15:37,766 WARNING awx.main.dispatch scaling up worker pid:204
2021-04-13 11:18:28,301 DEBUG awx.main.dispatch delivered 75b10717-4ccb-4612-8546-50b7c5ed2c9b to worker[204] qsize 0
2021-04-13 11:18:28,309 DEBUG awx.main.dispatch delivered 842f3c00-6ec1-417f-9234-b0e9c9aaba49 to worker[202] qsize 0
2021-04-13 11:18:28,310 DEBUG awx.main.dispatch delivered 7dab6cc1-7e90-42ab-99a5-b7880cd9b575 to worker[183] qsize 0
2021-04-13 11:18:28,309 DEBUG awx.main.dispatch task 75b10717-4ccb-4612-8546-50b7c5ed2c9b starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:18:28,316 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:18:28,312 DEBUG awx.main.dispatch delivered 8e3f2b20-9206-45fa-9e37-42a467d3b8b1 to worker[192] qsize 0
2021-04-13 11:18:28,314 DEBUG awx.main.dispatch task 842f3c00-6ec1-417f-9234-b0e9c9aaba49 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:18:28,318 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:18:28,318 DEBUG awx.main.dispatch delivered efa5c8d8-f11c-421f-8034-ea9646d3c58b to worker[195] qsize 0
2021-04-13 11:18:28,314 DEBUG awx.main.dispatch task 7dab6cc1-7e90-42ab-99a5-b7880cd9b575 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:18:28,320 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:18:28,320 DEBUG awx.main.dispatch task 8e3f2b20-9206-45fa-9e37-42a467d3b8b1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:18:28,322 DEBUG awx.main.dispatch task efa5c8d8-f11c-421f-8034-ea9646d3c58b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:18:28,327 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:18:28,338 WARNING awx.main.dispatch scaling up worker pid:207
2021-04-13 11:18:28,344 DEBUG awx.main.dispatch delivered fadb7c9d-6b09-49d6-a11f-d43d84cb7323 to worker[207] qsize 0
2021-04-13 11:18:28,347 DEBUG awx.main.dispatch task fadb7c9d-6b09-49d6-a11f-d43d84cb7323 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:18:28,355 WARNING awx.main.dispatch scaling up worker pid:209
2021-04-13 11:18:28,359 DEBUG awx.main.dispatch delivered f397950f-bf60-4ec8-9621-84971c663c35 to worker[209] qsize 0
2021-04-13 11:18:28,363 DEBUG awx.main.dispatch task f397950f-bf60-4ec8-9621-84971c663c35 starting awx.main.tasks.awx_k8s_reaper(*[])
RESULT 2
OKREADY
RESULT 2
OKREADY
2021-04-13 11:19:35,079 DEBUG awx.main.dispatch task 919f78d8-ecab-4638-a4d5-dbedf2fe0c11 is finished
2021-04-13 11:20:10,771 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2021-04-13 11:20:10,787 DEBUG awx.main.dispatch task 842f3c00-6ec1-417f-9234-b0e9c9aaba49 is finished
2021-04-13 11:20:15,350 DEBUG awx.main.scheduler Starting Scheduler
2021-04-13 11:20:18,799 DEBUG awx.main.dispatch task ddd66af1-823b-4918-8f43-d61d3f462f8a is finished
2021-04-13 11:16:49,101 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f950b9b7-ab6d-4b19-8108-cd8f9d9ce610, queue=dev-awx-0)
[2021-04-13 11:16:49,101: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f950b9b7-ab6d-4b19-8108-cd8f9d9ce610, queue=dev-awx-0)
2021-04-13 11:20:23,603 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8850ddc9-bbf1-42b0-88ae-f7c12d05f0be, queue=dev-awx-0)
[2021-04-13 11:20:23,603: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8850ddc9-bbf1-42b0-88ae-f7c12d05f0be, queue=dev-awx-0)
2021-04-13 11:20:23,620 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(71e506c6-4a58-4882-93b2-d5fd83855515, queue=awx_private_queue)
[2021-04-13 11:20:23,620: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(71e506c6-4a58-4882-93b2-d5fd83855515, queue=awx_private_queue)
2021-04-13 11:20:23,638 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1963b823-d6dc-4bd9-9e2b-5967b93940ae, queue=awx_private_queue)
[2021-04-13 11:20:23,638: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1963b823-d6dc-4bd9-9e2b-5967b93940ae, queue=awx_private_queue)
2021-04-13 11:20:24,182 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:20:24,404 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-04-13 11:20:24,405 DEBUG awx.main.dispatch task 8e3f2b20-9206-45fa-9e37-42a467d3b8b1 is finished
2021-04-13 11:20:24,544 DEBUG awx.main.dispatch task f397950f-bf60-4ec8-9621-84971c663c35 is finished
2021-04-13 11:20:30,259 DEBUG awx.main.tasks Last scheduler run was: 2021-04-13 11:16:27.130818+00:00
2021-04-13 11:20:30,748 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-13 11:20:30,749 DEBUG awx.main.dispatch task efa5c8d8-f11c-421f-8034-ea9646d3c58b is finished
2021-04-13 11:20:44,065 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e21070c4-f3ec-4c3d-a220-5f3507852d00, queue=awx_private_queue)
[2021-04-13 11:20:44,065: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e21070c4-f3ec-4c3d-a220-5f3507852d00, queue=awx_private_queue)
2021-04-13 11:20:51,854 DEBUG awx.main.dispatch task 7dab6cc1-7e90-42ab-99a5-b7880cd9b575 is finished
2021-04-13 11:20:53,638 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3b8505b9-2373-4613-a4c0-dc4fc2a79b84, queue=awx_private_queue)
[2021-04-13 11:20:53,638: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3b8505b9-2373-4613-a4c0-dc4fc2a79b84, queue=awx_private_queue)
RESULT 2
OKREADY
2021-04-13 11:21:04,065 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e966cbee-3db0-4ef2-a4c8-b3b5f90c0788, queue=awx_private_queue)
[2021-04-13 11:21:04,065: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e966cbee-3db0-4ef2-a4c8-b3b5f90c0788, queue=awx_private_queue)
2021-04-13 11:21:04,085 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(9def1c4d-0b6d-4e09-9698-45dd0d0501cf, queue=dev-awx-0)
[2021-04-13 11:21:04,085: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(9def1c4d-0b6d-4e09-9698-45dd0d0501cf, queue=dev-awx-0)
2021-04-13 11:21:07,239 DEBUG awx.main.dispatch task fadb7c9d-6b09-49d6-a11f-d43d84cb7323 is finished
2021-04-13 11:21:07,916 DEBUG awx.main.dispatch task 75b10717-4ccb-4612-8546-50b7c5ed2c9b is finished
2021-04-13 11:21:23,614 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(bd214a41-e8a1-4d75-8ce1-cb136e32c3c6, queue=dev-awx-0)
[2021-04-13 11:21:23,614: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(bd214a41-e8a1-4d75-8ce1-cb136e32c3c6, queue=dev-awx-0)
2021-04-13 11:21:23,751 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(36de3545-8bae-4dd9-85a7-65ff942e0b1e, queue=awx_private_queue)
[2021-04-13 11:21:23,751: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(36de3545-8bae-4dd9-85a7-65ff942e0b1e, queue=awx_private_queue)
2021-04-13 11:21:24,066 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b78385b6-c9a8-4d29-9330-6d0c5b2187df, queue=awx_private_queue)
[2021-04-13 11:21:24,066: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b78385b6-c9a8-4d29-9330-6d0c5b2187df, queue=awx_private_queue)
2021-04-13 11:21:19,873 DEBUG awx.main.dispatch delivered 1dfa5967-e522-4017-850b-dc01eee9836c to worker[202] qsize 0
2021-04-13 11:21:33,627 DEBUG awx.main.dispatch delivered d0d6bed8-2e87-4a8f-b692-dbe4c5a60c67 to worker[192] qsize 0
2021-04-13 11:21:33,631 DEBUG awx.main.dispatch delivered 91fbc748-2f7e-445c-8cd0-c625384fa85f to worker[195] qsize 0
2021-04-13 11:21:33,633 DEBUG awx.main.dispatch delivered e59122f5-c93a-4c1b-b6a9-9adab04e008a to worker[198] qsize 0
2021-04-13 11:21:33,629 DEBUG awx.main.dispatch task 1dfa5967-e522-4017-850b-dc01eee9836c starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:21:33,635 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:21:33,635 DEBUG awx.main.dispatch delivered bf221498-3b81-4a08-a31a-039a3d940ed3 to worker[204] qsize 0
2021-04-13 11:21:33,637 DEBUG awx.main.dispatch delivered 6f5f8c03-36a2-42e1-81c7-9793351473af to worker[200] qsize 0
2021-04-13 11:21:33,638 DEBUG awx.main.dispatch delivered f950b9b7-ab6d-4b19-8108-cd8f9d9ce610 to worker[209] qsize 0
2021-04-13 11:21:33,635 DEBUG awx.main.dispatch task 91fbc748-2f7e-445c-8cd0-c625384fa85f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:21:33,641 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:21:33,640 WARNING awx.main.dispatch scaling down worker pid:183
2021-04-13 11:21:33,642 WARNING awx.main.dispatch scaling down worker pid:207
2021-04-13 11:21:33,633 DEBUG awx.main.dispatch task d0d6bed8-2e87-4a8f-b692-dbe4c5a60c67 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:21:33,646 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:21:33,637 DEBUG awx.main.dispatch task e59122f5-c93a-4c1b-b6a9-9adab04e008a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:21:33,643 DEBUG awx.main.dispatch task 6f5f8c03-36a2-42e1-81c7-9793351473af starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:21:33,647 WARNING awx.main.dispatch worker exiting gracefully pid:207
2021-04-13 11:21:33,646 DEBUG awx.main.dispatch task bf221498-3b81-4a08-a31a-039a3d940ed3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:21:33,652 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:21:33,647 WARNING awx.main.dispatch worker exiting gracefully pid:183
2021-04-13 11:21:33,659 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:21:33,643 DEBUG awx.main.dispatch task f950b9b7-ab6d-4b19-8108-cd8f9d9ce610 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:21:33,659 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:21:33,665 DEBUG awx.main.dispatch task 91fbc748-2f7e-445c-8cd0-c625384fa85f is finished
2021-04-13 11:21:33,665 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-04-13 11:21:33,669 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:21:33,664 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:21:33,669 DEBUG awx.main.dispatch task 1dfa5967-e522-4017-850b-dc01eee9836c is finished
2021-04-13 11:21:33,670 DEBUG awx.main.dispatch task d0d6bed8-2e87-4a8f-b692-dbe4c5a60c67 is finished
2021-04-13 11:21:33,664 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:21:33,669 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:21:33,673 DEBUG awx.main.dispatch task 6f5f8c03-36a2-42e1-81c7-9793351473af is finished
2021-04-13 11:21:33,675 DEBUG awx.main.dispatch task bf221498-3b81-4a08-a31a-039a3d940ed3 is finished
2021-04-13 11:21:33,675 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:21:33,681 DEBUG awx.main.dispatch task e59122f5-c93a-4c1b-b6a9-9adab04e008a is finished
2021-04-13 11:21:33,676 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140082896053824_2" does not exist
2021-04-13 11:21:33,683 DEBUG awx.main.dispatch task f950b9b7-ab6d-4b19-8108-cd8f9d9ce610 is finished
2021-04-13 11:21:44,067 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7bec09b3-cca2-4a5a-930c-7408032115ab, queue=awx_private_queue)
[2021-04-13 11:21:44,067: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7bec09b3-cca2-4a5a-930c-7408032115ab, queue=awx_private_queue)
2021-04-13 11:21:49,840 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(0275fe79-dba9-4b07-8821-66172d2f9eea, queue=dev-awx-0)
[2021-04-13 11:21:49,840: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(0275fe79-dba9-4b07-8821-66172d2f9eea, queue=dev-awx-0)
2021-04-13 11:21:53,751 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(78dc1a46-bad9-4055-873c-9941b1658aee, queue=awx_private_queue)
[2021-04-13 11:21:53,751: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(78dc1a46-bad9-4055-873c-9941b1658aee, queue=awx_private_queue)
RESULT 2
OKREADY
2021-04-13 11:22:04,066 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(897567b2-671a-4543-96ed-636ddea782a7, queue=awx_private_queue)
[2021-04-13 11:22:04,066: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(897567b2-671a-4543-96ed-636ddea782a7, queue=awx_private_queue)
2021-04-13 11:22:04,090 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(6eb3db03-1ed5-4346-acd3-d395fae02fef, queue=dev-awx-0)
[2021-04-13 11:22:04,090: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(6eb3db03-1ed5-4346-acd3-d395fae02fef, queue=dev-awx-0)
2021-04-13 11:22:23,626 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8431b0bb-d6f8-403b-a0ea-05f7b38a066a, queue=dev-awx-0)
[2021-04-13 11:22:23,626: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8431b0bb-d6f8-403b-a0ea-05f7b38a066a, queue=dev-awx-0)
2021-04-13 11:22:23,751 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1bf16e8e-ad95-4a97-bd5d-1d8c894fcf9a, queue=awx_private_queue)
[2021-04-13 11:22:23,751: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1bf16e8e-ad95-4a97-bd5d-1d8c894fcf9a, queue=awx_private_queue)
2021-04-13 11:22:24,066 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(22105579-df7a-43de-9e1e-a32cc9a74834, queue=awx_private_queue)
[2021-04-13 11:22:24,066: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(22105579-df7a-43de-9e1e-a32cc9a74834, queue=awx_private_queue)
2021-04-13 11:22:28,751 DEBUG awx.main.dispatch delivered 8850ddc9-bbf1-42b0-88ae-f7c12d05f0be to worker[202] qsize 0
2021-04-13 11:22:28,757 DEBUG awx.main.dispatch delivered 71e506c6-4a58-4882-93b2-d5fd83855515 to worker[198] qsize 0
2021-04-13 11:22:28,759 DEBUG awx.main.dispatch delivered 1963b823-d6dc-4bd9-9e2b-5967b93940ae to worker[200] qsize 0
2021-04-13 11:22:28,760 DEBUG awx.main.dispatch delivered e21070c4-f3ec-4c3d-a220-5f3507852d00 to worker[192] qsize 0
2021-04-13 11:22:28,762 DEBUG awx.main.dispatch delivered 3b8505b9-2373-4613-a4c0-dc4fc2a79b84 to worker[209] qsize 0
2021-04-13 11:22:28,763 DEBUG awx.main.dispatch delivered e966cbee-3db0-4ef2-a4c8-b3b5f90c0788 to worker[204] qsize 0
2021-04-13 11:22:28,762 DEBUG awx.main.dispatch task 8850ddc9-bbf1-42b0-88ae-f7c12d05f0be starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:22:28,765 DEBUG awx.main.dispatch delivered 9def1c4d-0b6d-4e09-9698-45dd0d0501cf to worker[195] qsize 0
2021-04-13 11:22:28,766 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:22:28,764 DEBUG awx.main.dispatch task 71e506c6-4a58-4882-93b2-d5fd83855515 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:22:28,769 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:22:28,765 DEBUG awx.main.dispatch task e21070c4-f3ec-4c3d-a220-5f3507852d00 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:22:28,770 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:22:28,762 DEBUG awx.main.dispatch task 1963b823-d6dc-4bd9-9e2b-5967b93940ae starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:22:28,769 DEBUG awx.main.dispatch task 9def1c4d-0b6d-4e09-9698-45dd0d0501cf starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:22:28,770 DEBUG awx.main.dispatch task 3b8505b9-2373-4613-a4c0-dc4fc2a79b84 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:22:28,768 DEBUG awx.main.dispatch task e966cbee-3db0-4ef2-a4c8-b3b5f90c0788 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:22:28,779 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:22:28,782 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-04-13 11:22:28,784 DEBUG awx.main.dispatch task 8850ddc9-bbf1-42b0-88ae-f7c12d05f0be is finished
2021-04-13 11:22:28,788 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:22:28,781 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:22:28,789 DEBUG awx.main.dispatch task e21070c4-f3ec-4c3d-a220-5f3507852d00 is finished
2021-04-13 11:22:28,785 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140082896053824_4" does not exist
2021-04-13 11:22:28,787 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:22:28,791 DEBUG awx.main.dispatch task 9def1c4d-0b6d-4e09-9698-45dd0d0501cf is finished
2021-04-13 11:22:28,793 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:22:28,792 DEBUG awx.main.dispatch task 71e506c6-4a58-4882-93b2-d5fd83855515 is finished
2021-04-13 11:22:28,794 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:22:28,796 DEBUG awx.main.dispatch task e966cbee-3db0-4ef2-a4c8-b3b5f90c0788 is finished
2021-04-13 11:22:28,792 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:22:28,798 DEBUG awx.main.dispatch task 1963b823-d6dc-4bd9-9e2b-5967b93940ae is finished
2021-04-13 11:22:28,799 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:22:28,801 DEBUG awx.main.dispatch task 3b8505b9-2373-4613-a4c0-dc4fc2a79b84 is finished
2021-04-13 11:22:30,042 DEBUG awx.main.dispatch delivered bd214a41-e8a1-4d75-8ce1-cb136e32c3c6 to worker[204] qsize 0
2021-04-13 11:22:30,044 DEBUG awx.main.dispatch delivered 36de3545-8bae-4dd9-85a7-65ff942e0b1e to worker[202] qsize 0
2021-04-13 11:22:30,045 DEBUG awx.main.dispatch task bd214a41-e8a1-4d75-8ce1-cb136e32c3c6 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-13 11:22:30,046 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-13 11:22:30,046 DEBUG awx.main.dispatch delivered b78385b6-c9a8-4d29-9330-6d0c5b2187df to worker[209] qsize 0
2021-04-13 11:22:30,048 DEBUG awx.main.dispatch delivered 7bec09b3-cca2-4a5a-930c-7408032115ab to worker[200] qsize 0
2021-04-13 11:22:30,047 DEBUG awx.main.dispatch task 36de3545-8bae-4dd9-85a7-65ff942e0b1e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:22:30,049 DEBUG awx.main.dispatch delivered 0275fe79-dba9-4b07-8821-66172d2f9eea to worker[192] qsize 0
2021-04-13 11:22:30,052 DEBUG awx.main.dispatch task 0275fe79-dba9-4b07-8821-66172d2f9eea starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-04-13 11:22:30,053 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-04-13 11:22:30,050 DEBUG awx.main.dispatch task b78385b6-c9a8-4d29-9330-6d0c5b2187df starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:22:30,053 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:22:30,052 DEBUG awx.main.dispatch delivered 78dc1a46-bad9-4055-873c-9941b1658aee to worker[198] qsize 0
2021-04-13 11:22:30,052 DEBUG awx.main.dispatch task 7bec09b3-cca2-4a5a-930c-7408032115ab starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:22:30,055 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:22:30,056 DEBUG awx.main.dispatch delivered 897567b2-671a-4543-96ed-636ddea782a7 to worker[195] qsize 0
2021-04-13 11:22:30,055 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-04-13 11:22:30,057 DEBUG awx.main.dispatch task 78dc1a46-bad9-4055-873c-9941b1658aee starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:22:30,058 DEBUG awx.main.dispatch task bd214a41-e8a1-4d75-8ce1-cb136e32c3c6 is finished
2021-04-13 11:22:30,059 DEBUG awx.main.dispatch task 897567b2-671a-4543-96ed-636ddea782a7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:22:30,060 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:22:30,061 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:22:30,064 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:22:30,066 DEBUG awx.main.dispatch task 36de3545-8bae-4dd9-85a7-65ff942e0b1e is finished
2021-04-13 11:22:30,069 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:22:30,068 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:22:30,071 DEBUG awx.main.dispatch task b78385b6-c9a8-4d29-9330-6d0c5b2187df is finished
2021-04-13 11:22:30,074 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:22:30,074 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:22:30,074 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:22:30,076 DEBUG awx.main.dispatch task 7bec09b3-cca2-4a5a-930c-7408032115ab is finished
2021-04-13 11:22:30,076 DEBUG awx.main.dispatch task 897567b2-671a-4543-96ed-636ddea782a7 is finished
2021-04-13 11:22:30,072 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 470, in awx_isolated_heartbeat
for isolated_instance in isolated_instance_qs:
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-04-13 11:22:30,076 DEBUG awx.main.dispatch task 78dc1a46-bad9-4055-873c-9941b1658aee is finished
2021-04-13 11:22:30,077 DEBUG awx.main.dispatch task 0275fe79-dba9-4b07-8821-66172d2f9eea is finished
2021-04-13 11:22:30,076 WARNING awx.main.dispatch scaling up worker pid:216
2021-04-13 11:22:30,081 DEBUG awx.main.dispatch delivered 6eb3db03-1ed5-4346-acd3-d395fae02fef to worker[216] qsize 0
2021-04-13 11:22:30,085 WARNING awx.main.dispatch scaling down worker pid:192
2021-04-13 11:22:30,084 DEBUG awx.main.dispatch task 6eb3db03-1ed5-4346-acd3-d395fae02fef starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:22:30,086 WARNING awx.main.dispatch scaling down worker pid:195
2021-04-13 11:22:30,087 WARNING awx.main.dispatch worker exiting gracefully pid:192
2021-04-13 11:22:30,088 WARNING awx.main.dispatch worker exiting gracefully pid:195
2021-04-13 11:22:30,088 WARNING awx.main.dispatch scaling down worker pid:198
2021-04-13 11:22:30,090 WARNING awx.main.dispatch scaling down worker pid:200
2021-04-13 11:22:30,090 WARNING awx.main.dispatch worker exiting gracefully pid:198
2021-04-13 11:22:30,092 WARNING awx.main.dispatch worker exiting gracefully pid:200
2021-04-13 11:22:30,103 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140082896053824_1" does not exist
2021-04-13 11:22:30,116 DEBUG awx.main.dispatch task 6eb3db03-1ed5-4346-acd3-d395fae02fef is finished
2021-04-13 11:22:30,112 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-13 11:22:30,151 DEBUG awx.main.dispatch delivered 1bf16e8e-ad95-4a97-bd5d-1d8c894fcf9a to worker[204] qsize 0
2021-04-13 11:22:30,159 DEBUG awx.main.dispatch delivered 22105579-df7a-43de-9e1e-a32cc9a74834 to worker[202] qsize 0
2021-04-13 11:22:30,162 DEBUG awx.main.dispatch task 22105579-df7a-43de-9e1e-a32cc9a74834 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:22:30,171 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:22:30,171 DEBUG awx.main.dispatch task 1bf16e8e-ad95-4a97-bd5d-1d8c894fcf9a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:22:30,182 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:22:30,180 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:22:30,183 DEBUG awx.main.dispatch task 22105579-df7a-43de-9e1e-a32cc9a74834 is finished
2021-04-13 11:22:30,185 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:22:30,187 DEBUG awx.main.dispatch task 1bf16e8e-ad95-4a97-bd5d-1d8c894fcf9a is finished
2021-04-13 11:22:44,076 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(670d927c-9600-4e05-8d45-f9589efc06ca, queue=awx_private_queue)
[2021-04-13 11:22:44,076: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(670d927c-9600-4e05-8d45-f9589efc06ca, queue=awx_private_queue)
2021-04-13 11:22:44,115 DEBUG awx.main.dispatch delivered 670d927c-9600-4e05-8d45-f9589efc06ca to worker[204] qsize 0
2021-04-13 11:22:44,161 DEBUG awx.main.dispatch task 670d927c-9600-4e05-8d45-f9589efc06ca starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:22:44,181 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:22:44,183 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:22:44,189 DEBUG awx.main.dispatch task 670d927c-9600-4e05-8d45-f9589efc06ca is finished
2021-04-13 11:22:53,758 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(40bfc44e-0e44-49f0-b41b-558928a548ca, queue=awx_private_queue)
[2021-04-13 11:22:53,758: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(40bfc44e-0e44-49f0-b41b-558928a548ca, queue=awx_private_queue)
2021-04-13 11:22:53,799 DEBUG awx.main.dispatch delivered 40bfc44e-0e44-49f0-b41b-558928a548ca to worker[209] qsize 0
2021-04-13 11:22:53,814 DEBUG awx.main.dispatch task 40bfc44e-0e44-49f0-b41b-558928a548ca starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:22:53,844 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:22:53,846 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:22:53,851 DEBUG awx.main.dispatch task 40bfc44e-0e44-49f0-b41b-558928a548ca is finished
RESULT 2
OKREADY
2021-04-13 11:23:04,094 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(1b5b9e6a-d9e2-44d8-a996-417af72b243e, queue=dev-awx-0)
[2021-04-13 11:23:04,094: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(1b5b9e6a-d9e2-44d8-a996-417af72b243e, queue=dev-awx-0)
2021-04-13 11:23:04,130 DEBUG awx.main.dispatch delivered 1b5b9e6a-d9e2-44d8-a996-417af72b243e to worker[216] qsize 0
2021-04-13 11:23:04,132 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(320b7dd6-d78e-4735-bf6c-ebc7668ce54f, queue=awx_private_queue)
[2021-04-13 11:23:04,132: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(320b7dd6-d78e-4735-bf6c-ebc7668ce54f, queue=awx_private_queue)
2021-04-13 11:23:04,160 DEBUG awx.main.dispatch delivered 320b7dd6-d78e-4735-bf6c-ebc7668ce54f to worker[209] qsize 0
2021-04-13 11:23:04,145 DEBUG awx.main.dispatch task 1b5b9e6a-d9e2-44d8-a996-417af72b243e starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:23:04,166 DEBUG awx.main.dispatch task 320b7dd6-d78e-4735-bf6c-ebc7668ce54f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:23:04,180 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:23:04,176 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140082896053824_2" does not exist
2021-04-13 11:23:04,181 DEBUG awx.main.dispatch task 1b5b9e6a-d9e2-44d8-a996-417af72b243e is finished
2021-04-13 11:23:04,182 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:23:04,187 DEBUG awx.main.dispatch task 320b7dd6-d78e-4735-bf6c-ebc7668ce54f is finished
2021-04-13 11:23:23,624 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d8ffa66c-2b6f-41d0-b25f-a97201fc9b6e, queue=dev-awx-0)
[2021-04-13 11:23:23,624: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d8ffa66c-2b6f-41d0-b25f-a97201fc9b6e, queue=dev-awx-0)
2021-04-13 11:23:23,665 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-13 11:23:23,754 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(01a5dbc0-5113-475d-8576-a6c758d9ff39, queue=awx_private_queue)
[2021-04-13 11:23:23,754: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(01a5dbc0-5113-475d-8576-a6c758d9ff39, queue=awx_private_queue)
2021-04-13 11:23:23,783 DEBUG awx.main.dispatch delivered 01a5dbc0-5113-475d-8576-a6c758d9ff39 to worker[209] qsize 0
2021-04-13 11:23:23,792 DEBUG awx.main.dispatch task 01a5dbc0-5113-475d-8576-a6c758d9ff39 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:23:23,810 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:23:23,813 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:23:23,817 DEBUG awx.main.dispatch task 01a5dbc0-5113-475d-8576-a6c758d9ff39 is finished
2021-04-13 11:23:24,131 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e5afe3ad-254a-43b7-ae88-47536ad68a29, queue=awx_private_queue)
[2021-04-13 11:23:24,131: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e5afe3ad-254a-43b7-ae88-47536ad68a29, queue=awx_private_queue)
2021-04-13 11:23:24,563 DEBUG awx.main.dispatch delivered e5afe3ad-254a-43b7-ae88-47536ad68a29 to worker[204] qsize 0
2021-04-13 11:23:24,579 DEBUG awx.main.dispatch task e5afe3ad-254a-43b7-ae88-47536ad68a29 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:23:24,611 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:23:24,613 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:23:24,618 DEBUG awx.main.dispatch task e5afe3ad-254a-43b7-ae88-47536ad68a29 is finished
2021-04-13 11:23:44,898 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f4af7ab8-c0a5-45be-b879-cc66fe4afeac, queue=awx_private_queue)
[2021-04-13 11:23:44,898: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f4af7ab8-c0a5-45be-b879-cc66fe4afeac, queue=awx_private_queue)
2021-04-13 11:23:44,939 DEBUG awx.main.dispatch delivered f4af7ab8-c0a5-45be-b879-cc66fe4afeac to worker[202] qsize 0
2021-04-13 11:23:44,962 DEBUG awx.main.dispatch task f4af7ab8-c0a5-45be-b879-cc66fe4afeac starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:23:45,132 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:23:45,134 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:23:45,139 DEBUG awx.main.dispatch task f4af7ab8-c0a5-45be-b879-cc66fe4afeac is finished
2021-04-13 11:23:53,754 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(903d472b-bf8a-488b-b6cd-3c251f3f329f, queue=awx_private_queue)
[2021-04-13 11:23:53,754: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(903d472b-bf8a-488b-b6cd-3c251f3f329f, queue=awx_private_queue)
2021-04-13 11:23:53,781 DEBUG awx.main.dispatch delivered 903d472b-bf8a-488b-b6cd-3c251f3f329f to worker[216] qsize 0
2021-04-13 11:23:53,791 DEBUG awx.main.dispatch task 903d472b-bf8a-488b-b6cd-3c251f3f329f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:23:53,817 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:23:53,821 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:23:54,012 DEBUG awx.main.dispatch task 903d472b-bf8a-488b-b6cd-3c251f3f329f is finished
RESULT 2
OKREADY
2021-04-13 11:24:04,096 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8d098e46-775a-400c-8d24-f8a162e17ad8, queue=dev-awx-0)
[2021-04-13 11:24:04,096: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8d098e46-775a-400c-8d24-f8a162e17ad8, queue=dev-awx-0)
2021-04-13 11:24:04,173 DEBUG awx.main.dispatch delivered 8d098e46-775a-400c-8d24-f8a162e17ad8 to worker[204] qsize 0
2021-04-13 11:24:04,195 DEBUG awx.main.dispatch task 8d098e46-775a-400c-8d24-f8a162e17ad8 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:24:04,222 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140082896053824_1" does not exist
2021-04-13 11:24:04,229 DEBUG awx.main.dispatch task 8d098e46-775a-400c-8d24-f8a162e17ad8 is finished
2021-04-13 11:24:04,894 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(20f7c683-bd29-48a9-b7c9-dcc3c16dd3d0, queue=awx_private_queue)
[2021-04-13 11:24:04,894: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(20f7c683-bd29-48a9-b7c9-dcc3c16dd3d0, queue=awx_private_queue)
2021-04-13 11:24:04,922 DEBUG awx.main.dispatch delivered 20f7c683-bd29-48a9-b7c9-dcc3c16dd3d0 to worker[209] qsize 0
2021-04-13 11:24:04,932 DEBUG awx.main.dispatch task 20f7c683-bd29-48a9-b7c9-dcc3c16dd3d0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:24:04,959 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:24:04,961 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:24:04,967 DEBUG awx.main.dispatch task 20f7c683-bd29-48a9-b7c9-dcc3c16dd3d0 is finished
2021-04-13 11:24:23,635 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(2aafa716-9039-470c-ad24-0c7197835205, queue=dev-awx-0)
[2021-04-13 11:24:23,635: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(2aafa716-9039-470c-ad24-0c7197835205, queue=dev-awx-0)
2021-04-13 11:24:23,694 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-13 11:24:23,756 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ddba56ea-d08f-4e2c-96d0-11179e072338, queue=awx_private_queue)
[2021-04-13 11:24:23,756: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ddba56ea-d08f-4e2c-96d0-11179e072338, queue=awx_private_queue)
2021-04-13 11:24:23,795 DEBUG awx.main.dispatch delivered ddba56ea-d08f-4e2c-96d0-11179e072338 to worker[204] qsize 0
2021-04-13 11:24:23,802 DEBUG awx.main.dispatch task ddba56ea-d08f-4e2c-96d0-11179e072338 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:24:23,817 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:24:23,820 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:24:23,825 DEBUG awx.main.dispatch task ddba56ea-d08f-4e2c-96d0-11179e072338 is finished
2021-04-13 11:24:24,895 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8fd9d8a7-1160-466f-aa3e-75518998ab85, queue=awx_private_queue)
[2021-04-13 11:24:24,895: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8fd9d8a7-1160-466f-aa3e-75518998ab85, queue=awx_private_queue)
2021-04-13 11:24:24,922 DEBUG awx.main.dispatch delivered 8fd9d8a7-1160-466f-aa3e-75518998ab85 to worker[204] qsize 0
2021-04-13 11:24:24,930 DEBUG awx.main.dispatch task 8fd9d8a7-1160-466f-aa3e-75518998ab85 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:24:24,943 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:24:24,945 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:24:24,955 DEBUG awx.main.dispatch task 8fd9d8a7-1160-466f-aa3e-75518998ab85 is finished
2021-04-13 11:24:44,896 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(81536792-81c0-4ef0-946a-84d4d8b17484, queue=awx_private_queue)
[2021-04-13 11:24:44,896: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(81536792-81c0-4ef0-946a-84d4d8b17484, queue=awx_private_queue)
2021-04-13 11:24:44,934 DEBUG awx.main.dispatch delivered 81536792-81c0-4ef0-946a-84d4d8b17484 to worker[204] qsize 0
2021-04-13 11:24:44,945 DEBUG awx.main.dispatch task 81536792-81c0-4ef0-946a-84d4d8b17484 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:24:44,960 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:24:44,962 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:24:44,967 DEBUG awx.main.dispatch task 81536792-81c0-4ef0-946a-84d4d8b17484 is finished
2021-04-13 11:24:53,765 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(fa447df4-4fc8-4c69-b822-17072278aae1, queue=awx_private_queue)
[2021-04-13 11:24:53,765: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(fa447df4-4fc8-4c69-b822-17072278aae1, queue=awx_private_queue)
2021-04-13 11:24:53,796 DEBUG awx.main.dispatch delivered fa447df4-4fc8-4c69-b822-17072278aae1 to worker[216] qsize 0
2021-04-13 11:24:53,810 DEBUG awx.main.dispatch task fa447df4-4fc8-4c69-b822-17072278aae1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:24:53,831 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:24:53,833 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:24:53,838 DEBUG awx.main.dispatch task fa447df4-4fc8-4c69-b822-17072278aae1 is finished
RESULT 2
OKREADY
2021-04-13 11:25:04,093 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3531663a-3767-4a95-bcc0-d1339df28b61, queue=dev-awx-0)
[2021-04-13 11:25:04,093: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3531663a-3767-4a95-bcc0-d1339df28b61, queue=dev-awx-0)
2021-04-13 11:25:04,120 DEBUG awx.main.dispatch delivered 3531663a-3767-4a95-bcc0-d1339df28b61 to worker[204] qsize 0
2021-04-13 11:25:04,130 DEBUG awx.main.dispatch task 3531663a-3767-4a95-bcc0-d1339df28b61 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:25:04,153 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140082896053824_2" does not exist
2021-04-13 11:25:04,159 DEBUG awx.main.dispatch task 3531663a-3767-4a95-bcc0-d1339df28b61 is finished
2021-04-13 11:25:04,895 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(43dd4545-a9fd-439a-9467-6919968ab930, queue=awx_private_queue)
[2021-04-13 11:25:04,895: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(43dd4545-a9fd-439a-9467-6919968ab930, queue=awx_private_queue)
2021-04-13 11:25:04,931 DEBUG awx.main.dispatch delivered 43dd4545-a9fd-439a-9467-6919968ab930 to worker[216] qsize 0
2021-04-13 11:25:04,951 DEBUG awx.main.dispatch task 43dd4545-a9fd-439a-9467-6919968ab930 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:25:04,965 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:25:04,968 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:25:05,008 DEBUG awx.main.dispatch task 43dd4545-a9fd-439a-9467-6919968ab930 is finished
2021-04-13 11:25:23,645 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(034c3311-160b-4f75-9f2b-5784ce39bb8d, queue=dev-awx-0)
[2021-04-13 11:25:23,645: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(034c3311-160b-4f75-9f2b-5784ce39bb8d, queue=dev-awx-0)
2021-04-13 11:25:23,707 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-13 11:25:23,757 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e9eb78a7-dfc3-4607-9adf-a49ca999399b, queue=awx_private_queue)
[2021-04-13 11:25:23,757: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e9eb78a7-dfc3-4607-9adf-a49ca999399b, queue=awx_private_queue)
2021-04-13 11:25:23,793 DEBUG awx.main.dispatch delivered e9eb78a7-dfc3-4607-9adf-a49ca999399b to worker[204] qsize 0
2021-04-13 11:25:23,802 DEBUG awx.main.dispatch task e9eb78a7-dfc3-4607-9adf-a49ca999399b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:25:23,823 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:25:23,826 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:25:23,831 DEBUG awx.main.dispatch task e9eb78a7-dfc3-4607-9adf-a49ca999399b is finished
2021-04-13 11:25:24,894 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f495f4e0-8f3b-468f-8c41-5bbbef54d3cd, queue=awx_private_queue)
[2021-04-13 11:25:24,894: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f495f4e0-8f3b-468f-8c41-5bbbef54d3cd, queue=awx_private_queue)
2021-04-13 11:25:24,919 DEBUG awx.main.dispatch delivered f495f4e0-8f3b-468f-8c41-5bbbef54d3cd to worker[216] qsize 0
2021-04-13 11:25:24,927 DEBUG awx.main.dispatch task f495f4e0-8f3b-468f-8c41-5bbbef54d3cd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:25:24,943 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:25:24,945 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:25:24,950 DEBUG awx.main.dispatch task f495f4e0-8f3b-468f-8c41-5bbbef54d3cd is finished
2021-04-13 11:25:44,903 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(08b9900b-d436-41ef-9b40-3c7e8829f0c0, queue=awx_private_queue)
[2021-04-13 11:25:44,903: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(08b9900b-d436-41ef-9b40-3c7e8829f0c0, queue=awx_private_queue)
2021-04-13 11:25:44,933 DEBUG awx.main.dispatch delivered 08b9900b-d436-41ef-9b40-3c7e8829f0c0 to worker[204] qsize 0
2021-04-13 11:25:44,944 DEBUG awx.main.dispatch task 08b9900b-d436-41ef-9b40-3c7e8829f0c0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:25:44,965 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:25:44,967 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:25:44,971 DEBUG awx.main.dispatch task 08b9900b-d436-41ef-9b40-3c7e8829f0c0 is finished
2021-04-13 11:25:53,756 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0948cbf8-3d9c-430e-855d-eda50f7bb3d7, queue=awx_private_queue)
[2021-04-13 11:25:53,756: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0948cbf8-3d9c-430e-855d-eda50f7bb3d7, queue=awx_private_queue)
2021-04-13 11:25:53,784 DEBUG awx.main.dispatch delivered 0948cbf8-3d9c-430e-855d-eda50f7bb3d7 to worker[209] qsize 0
2021-04-13 11:25:53,798 DEBUG awx.main.dispatch task 0948cbf8-3d9c-430e-855d-eda50f7bb3d7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:25:53,824 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:25:53,827 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:25:53,833 DEBUG awx.main.dispatch task 0948cbf8-3d9c-430e-855d-eda50f7bb3d7 is finished
RESULT 2
OKREADY
2021-04-13 11:26:04,094 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(4843c528-5289-4330-bbde-dd4e9727395a, queue=dev-awx-0)
[2021-04-13 11:26:04,094: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(4843c528-5289-4330-bbde-dd4e9727395a, queue=dev-awx-0)
2021-04-13 11:26:04,122 DEBUG awx.main.dispatch delivered 4843c528-5289-4330-bbde-dd4e9727395a to worker[202] qsize 0
2021-04-13 11:26:04,135 DEBUG awx.main.dispatch task 4843c528-5289-4330-bbde-dd4e9727395a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:26:04,157 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140082896053824_2" does not exist
2021-04-13 11:26:04,162 DEBUG awx.main.dispatch task 4843c528-5289-4330-bbde-dd4e9727395a is finished
2021-04-13 11:26:04,902 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5b6a6db5-606c-4dde-ba05-e5062b95dc95, queue=awx_private_queue)
[2021-04-13 11:26:04,902: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5b6a6db5-606c-4dde-ba05-e5062b95dc95, queue=awx_private_queue)
2021-04-13 11:26:04,929 DEBUG awx.main.dispatch delivered 5b6a6db5-606c-4dde-ba05-e5062b95dc95 to worker[216] qsize 0
2021-04-13 11:26:04,938 DEBUG awx.main.dispatch task 5b6a6db5-606c-4dde-ba05-e5062b95dc95 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:26:04,964 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:26:04,966 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:26:04,971 DEBUG awx.main.dispatch task 5b6a6db5-606c-4dde-ba05-e5062b95dc95 is finished
2021-04-13 11:26:23,653 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(57ba4c46-7dba-4cd0-a13e-4c158843ce45, queue=dev-awx-0)
[2021-04-13 11:26:23,653: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(57ba4c46-7dba-4cd0-a13e-4c158843ce45, queue=dev-awx-0)
2021-04-13 11:26:23,716 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-13 11:26:23,755 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c033e623-fbe8-49a8-877e-7900d677d375, queue=awx_private_queue)
[2021-04-13 11:26:23,755: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c033e623-fbe8-49a8-877e-7900d677d375, queue=awx_private_queue)
2021-04-13 11:26:23,781 DEBUG awx.main.dispatch delivered c033e623-fbe8-49a8-877e-7900d677d375 to worker[209] qsize 0
2021-04-13 11:26:23,789 DEBUG awx.main.dispatch task c033e623-fbe8-49a8-877e-7900d677d375 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:26:23,806 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:26:23,808 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:26:23,813 DEBUG awx.main.dispatch task c033e623-fbe8-49a8-877e-7900d677d375 is finished
2021-04-13 11:26:24,902 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(80082f8f-c625-49ca-ac7e-b9eea436ff64, queue=awx_private_queue)
[2021-04-13 11:26:24,902: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(80082f8f-c625-49ca-ac7e-b9eea436ff64, queue=awx_private_queue)
2021-04-13 11:26:24,929 DEBUG awx.main.dispatch delivered 80082f8f-c625-49ca-ac7e-b9eea436ff64 to worker[209] qsize 0
2021-04-13 11:26:24,935 DEBUG awx.main.dispatch task 80082f8f-c625-49ca-ac7e-b9eea436ff64 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:26:24,948 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:26:24,950 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:26:24,956 DEBUG awx.main.dispatch task 80082f8f-c625-49ca-ac7e-b9eea436ff64 is finished
2021-04-13 11:26:44,999 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f75a506b-41c5-4856-b85b-30c9b098093b, queue=awx_private_queue)
[2021-04-13 11:26:44,999: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f75a506b-41c5-4856-b85b-30c9b098093b, queue=awx_private_queue)
2021-04-13 11:26:45,035 DEBUG awx.main.dispatch delivered f75a506b-41c5-4856-b85b-30c9b098093b to worker[209] qsize 0
2021-04-13 11:26:45,044 DEBUG awx.main.dispatch task f75a506b-41c5-4856-b85b-30c9b098093b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:26:45,057 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:26:45,059 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:26:45,067 DEBUG awx.main.dispatch task f75a506b-41c5-4856-b85b-30c9b098093b is finished
2021-04-13 11:26:53,756 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6acbdeb7-ad0e-4998-b370-82582008d88e, queue=awx_private_queue)
[2021-04-13 11:26:53,756: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6acbdeb7-ad0e-4998-b370-82582008d88e, queue=awx_private_queue)
2021-04-13 11:26:53,790 DEBUG awx.main.dispatch delivered 6acbdeb7-ad0e-4998-b370-82582008d88e to worker[216] qsize 0
2021-04-13 11:26:53,802 DEBUG awx.main.dispatch task 6acbdeb7-ad0e-4998-b370-82582008d88e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:26:53,820 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:26:53,823 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:26:53,828 DEBUG awx.main.dispatch task 6acbdeb7-ad0e-4998-b370-82582008d88e is finished
RESULT 2
OKREADY
2021-04-13 11:27:04,093 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a1b9ce18-990f-4d01-88c2-19ff875c3eb7, queue=dev-awx-0)
[2021-04-13 11:27:04,093: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a1b9ce18-990f-4d01-88c2-19ff875c3eb7, queue=dev-awx-0)
2021-04-13 11:27:04,130 DEBUG awx.main.dispatch delivered a1b9ce18-990f-4d01-88c2-19ff875c3eb7 to worker[209] qsize 0
2021-04-13 11:27:04,142 DEBUG awx.main.dispatch task a1b9ce18-990f-4d01-88c2-19ff875c3eb7 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:27:04,170 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140082896053824_3" does not exist
2021-04-13 11:27:04,175 DEBUG awx.main.dispatch task a1b9ce18-990f-4d01-88c2-19ff875c3eb7 is finished
2021-04-13 11:27:04,999 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(aa3dce6c-c190-484c-baa5-be01d3fc46bc, queue=awx_private_queue)
[2021-04-13 11:27:04,999: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(aa3dce6c-c190-484c-baa5-be01d3fc46bc, queue=awx_private_queue)
2021-04-13 11:27:05,033 DEBUG awx.main.dispatch delivered aa3dce6c-c190-484c-baa5-be01d3fc46bc to worker[209] qsize 0
2021-04-13 11:27:05,041 DEBUG awx.main.dispatch task aa3dce6c-c190-484c-baa5-be01d3fc46bc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:27:05,053 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:27:05,055 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:27:05,062 DEBUG awx.main.dispatch task aa3dce6c-c190-484c-baa5-be01d3fc46bc is finished
2021-04-13 11:27:23,658 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e0987441-600d-4dc5-871b-ce26a0452c6e, queue=dev-awx-0)
[2021-04-13 11:27:23,658: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e0987441-600d-4dc5-871b-ce26a0452c6e, queue=dev-awx-0)
2021-04-13 11:27:23,700 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-13 11:27:23,765 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e37ee1bb-5d59-47e9-9c18-c15006382bbe, queue=awx_private_queue)
[2021-04-13 11:27:23,765: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e37ee1bb-5d59-47e9-9c18-c15006382bbe, queue=awx_private_queue)
2021-04-13 11:27:23,810 DEBUG awx.main.dispatch delivered e37ee1bb-5d59-47e9-9c18-c15006382bbe to worker[216] qsize 0
2021-04-13 11:27:23,818 DEBUG awx.main.dispatch task e37ee1bb-5d59-47e9-9c18-c15006382bbe starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:27:23,837 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:27:23,840 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:27:23,844 DEBUG awx.main.dispatch task e37ee1bb-5d59-47e9-9c18-c15006382bbe is finished
2021-04-13 11:27:24,998 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1473e2b5-62db-4a1e-a085-de5080ab0c1f, queue=awx_private_queue)
[2021-04-13 11:27:24,998: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1473e2b5-62db-4a1e-a085-de5080ab0c1f, queue=awx_private_queue)
2021-04-13 11:27:25,029 DEBUG awx.main.dispatch delivered 1473e2b5-62db-4a1e-a085-de5080ab0c1f to worker[209] qsize 0
2021-04-13 11:27:25,036 DEBUG awx.main.dispatch task 1473e2b5-62db-4a1e-a085-de5080ab0c1f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:27:25,048 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:27:25,050 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:27:25,058 DEBUG awx.main.dispatch task 1473e2b5-62db-4a1e-a085-de5080ab0c1f is finished
2021-04-13 11:27:45,013 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ccf19ecd-82d6-4d29-9392-18b255a0ef83, queue=awx_private_queue)
[2021-04-13 11:27:45,013: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ccf19ecd-82d6-4d29-9392-18b255a0ef83, queue=awx_private_queue)
2021-04-13 11:27:45,063 DEBUG awx.main.dispatch delivered ccf19ecd-82d6-4d29-9392-18b255a0ef83 to worker[202] qsize 0
2021-04-13 11:27:45,074 DEBUG awx.main.dispatch task ccf19ecd-82d6-4d29-9392-18b255a0ef83 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:27:45,097 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:27:45,099 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:27:45,103 DEBUG awx.main.dispatch task ccf19ecd-82d6-4d29-9392-18b255a0ef83 is finished
2021-04-13 11:27:53,756 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f8fc1540-8c96-4664-a9b7-8e07f3354e8a, queue=awx_private_queue)
[2021-04-13 11:27:53,756: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f8fc1540-8c96-4664-a9b7-8e07f3354e8a, queue=awx_private_queue)
2021-04-13 11:27:53,796 DEBUG awx.main.dispatch delivered f8fc1540-8c96-4664-a9b7-8e07f3354e8a to worker[209] qsize 0
2021-04-13 11:27:53,804 DEBUG awx.main.dispatch task f8fc1540-8c96-4664-a9b7-8e07f3354e8a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:27:53,818 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:27:53,821 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:27:53,827 DEBUG awx.main.dispatch task f8fc1540-8c96-4664-a9b7-8e07f3354e8a is finished
RESULT 2
OKREADY
2021-04-13 11:28:04,093 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ce5c965c-ebed-408c-86ea-da8d8c3f6853, queue=dev-awx-0)
[2021-04-13 11:28:04,093: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ce5c965c-ebed-408c-86ea-da8d8c3f6853, queue=dev-awx-0)
2021-04-13 11:28:04,118 DEBUG awx.main.dispatch delivered ce5c965c-ebed-408c-86ea-da8d8c3f6853 to worker[216] qsize 0
2021-04-13 11:28:04,129 DEBUG awx.main.dispatch task ce5c965c-ebed-408c-86ea-da8d8c3f6853 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:28:04,146 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140082896053824_3" does not exist
2021-04-13 11:28:04,153 DEBUG awx.main.dispatch task ce5c965c-ebed-408c-86ea-da8d8c3f6853 is finished
2021-04-13 11:28:05,007 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5960272e-0b50-42d8-92c9-a9b3e9a632ca, queue=awx_private_queue)
[2021-04-13 11:28:05,007: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5960272e-0b50-42d8-92c9-a9b3e9a632ca, queue=awx_private_queue)
2021-04-13 11:28:05,033 DEBUG awx.main.dispatch delivered 5960272e-0b50-42d8-92c9-a9b3e9a632ca to worker[216] qsize 0
2021-04-13 11:28:05,043 DEBUG awx.main.dispatch task 5960272e-0b50-42d8-92c9-a9b3e9a632ca starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:28:05,055 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:28:05,057 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:28:05,064 DEBUG awx.main.dispatch task 5960272e-0b50-42d8-92c9-a9b3e9a632ca is finished
2021-04-13 11:28:23,666 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d48a9025-9ff4-402f-a318-116f16d81975, queue=dev-awx-0)
[2021-04-13 11:28:23,666: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d48a9025-9ff4-402f-a318-116f16d81975, queue=dev-awx-0)
2021-04-13 11:28:23,701 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-13 11:28:23,755 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(950dd4a2-4f99-4072-89ca-bc1387efcbd4, queue=awx_private_queue)
[2021-04-13 11:28:23,755: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(950dd4a2-4f99-4072-89ca-bc1387efcbd4, queue=awx_private_queue)
2021-04-13 11:28:23,782 DEBUG awx.main.dispatch delivered 950dd4a2-4f99-4072-89ca-bc1387efcbd4 to worker[204] qsize 0
2021-04-13 11:28:23,791 DEBUG awx.main.dispatch task 950dd4a2-4f99-4072-89ca-bc1387efcbd4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:28:23,812 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:28:23,815 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:28:23,820 DEBUG awx.main.dispatch task 950dd4a2-4f99-4072-89ca-bc1387efcbd4 is finished
2021-04-13 11:28:25,007 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a25347c2-d4fe-437b-ac40-683cf2eaa1fb, queue=awx_private_queue)
[2021-04-13 11:28:25,007: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a25347c2-d4fe-437b-ac40-683cf2eaa1fb, queue=awx_private_queue)
2021-04-13 11:28:25,041 DEBUG awx.main.dispatch delivered a25347c2-d4fe-437b-ac40-683cf2eaa1fb to worker[202] qsize 0
2021-04-13 11:28:25,052 DEBUG awx.main.dispatch task a25347c2-d4fe-437b-ac40-683cf2eaa1fb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:28:25,084 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:28:25,086 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:28:25,092 DEBUG awx.main.dispatch task a25347c2-d4fe-437b-ac40-683cf2eaa1fb is finished
2021-04-13 11:28:45,030 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fd36f9e2-de8a-4f90-b726-7c3bb02ece5c, queue=awx_private_queue)
[2021-04-13 11:28:45,030: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fd36f9e2-de8a-4f90-b726-7c3bb02ece5c, queue=awx_private_queue)
2021-04-13 11:28:45,081 DEBUG awx.main.dispatch delivered fd36f9e2-de8a-4f90-b726-7c3bb02ece5c to worker[216] qsize 0
2021-04-13 11:28:45,097 DEBUG awx.main.dispatch task fd36f9e2-de8a-4f90-b726-7c3bb02ece5c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:28:45,119 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:28:45,121 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:28:45,127 DEBUG awx.main.dispatch task fd36f9e2-de8a-4f90-b726-7c3bb02ece5c is finished
2021-04-13 11:28:53,759 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(71dfebfd-8698-458f-8551-a74933aef7cc, queue=awx_private_queue)
[2021-04-13 11:28:53,759: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(71dfebfd-8698-458f-8551-a74933aef7cc, queue=awx_private_queue)
2021-04-13 11:28:53,796 DEBUG awx.main.dispatch delivered 71dfebfd-8698-458f-8551-a74933aef7cc to worker[216] qsize 0
2021-04-13 11:28:53,807 DEBUG awx.main.dispatch task 71dfebfd-8698-458f-8551-a74933aef7cc starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:28:53,819 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:28:53,822 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:28:53,832 DEBUG awx.main.dispatch task 71dfebfd-8698-458f-8551-a74933aef7cc is finished
RESULT 2
OKREADY
2021-04-13 11:29:04,093 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ebf9ab50-fa41-4fde-8575-d69d8d3ce5d5, queue=dev-awx-0)
[2021-04-13 11:29:04,093: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ebf9ab50-fa41-4fde-8575-d69d8d3ce5d5, queue=dev-awx-0)
2021-04-13 11:29:04,122 DEBUG awx.main.dispatch delivered ebf9ab50-fa41-4fde-8575-d69d8d3ce5d5 to worker[216] qsize 0
2021-04-13 11:29:04,130 DEBUG awx.main.dispatch task ebf9ab50-fa41-4fde-8575-d69d8d3ce5d5 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:29:04,148 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140082896053824_4" does not exist
2021-04-13 11:29:04,152 DEBUG awx.main.dispatch task ebf9ab50-fa41-4fde-8575-d69d8d3ce5d5 is finished
2021-04-13 11:29:05,021 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(17c38133-8cd4-4fcc-9f79-fb9d59e774ec, queue=awx_private_queue)
[2021-04-13 11:29:05,021: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(17c38133-8cd4-4fcc-9f79-fb9d59e774ec, queue=awx_private_queue)
2021-04-13 11:29:05,051 DEBUG awx.main.dispatch delivered 17c38133-8cd4-4fcc-9f79-fb9d59e774ec to worker[216] qsize 0
2021-04-13 11:29:05,063 DEBUG awx.main.dispatch task 17c38133-8cd4-4fcc-9f79-fb9d59e774ec starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:29:05,076 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:29:05,078 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:29:05,085 DEBUG awx.main.dispatch task 17c38133-8cd4-4fcc-9f79-fb9d59e774ec is finished
2021-04-13 11:29:23,673 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(0254e1c4-2947-4ad6-aac0-67ea2282f94d, queue=dev-awx-0)
[2021-04-13 11:29:23,673: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(0254e1c4-2947-4ad6-aac0-67ea2282f94d, queue=dev-awx-0)
2021-04-13 11:29:23,713 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-13 11:29:23,756 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(aa37af01-3825-4116-ad85-c8991ff371a9, queue=awx_private_queue)
[2021-04-13 11:29:23,756: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(aa37af01-3825-4116-ad85-c8991ff371a9, queue=awx_private_queue)
2021-04-13 11:29:23,779 DEBUG awx.main.dispatch delivered aa37af01-3825-4116-ad85-c8991ff371a9 to worker[216] qsize 0
2021-04-13 11:29:23,787 DEBUG awx.main.dispatch task aa37af01-3825-4116-ad85-c8991ff371a9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:29:23,801 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:29:23,803 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:29:23,809 DEBUG awx.main.dispatch task aa37af01-3825-4116-ad85-c8991ff371a9 is finished
2021-04-13 11:29:25,020 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0b3f0f3d-7160-4956-872f-0f3a0b60ef33, queue=awx_private_queue)
[2021-04-13 11:29:25,020: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0b3f0f3d-7160-4956-872f-0f3a0b60ef33, queue=awx_private_queue)
2021-04-13 11:29:25,056 DEBUG awx.main.dispatch delivered 0b3f0f3d-7160-4956-872f-0f3a0b60ef33 to worker[202] qsize 0
2021-04-13 11:29:25,067 DEBUG awx.main.dispatch task 0b3f0f3d-7160-4956-872f-0f3a0b60ef33 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:29:25,090 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:29:25,092 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:29:25,096 DEBUG awx.main.dispatch task 0b3f0f3d-7160-4956-872f-0f3a0b60ef33 is finished
2021-04-13 11:29:45,046 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a5b7e802-85f8-4015-b7e1-ac773671325e, queue=awx_private_queue)
[2021-04-13 11:29:45,046: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a5b7e802-85f8-4015-b7e1-ac773671325e, queue=awx_private_queue)
2021-04-13 11:29:45,084 DEBUG awx.main.dispatch delivered a5b7e802-85f8-4015-b7e1-ac773671325e to worker[209] qsize 0
2021-04-13 11:29:45,096 DEBUG awx.main.dispatch task a5b7e802-85f8-4015-b7e1-ac773671325e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:29:45,118 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:29:45,120 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:29:45,124 DEBUG awx.main.dispatch task a5b7e802-85f8-4015-b7e1-ac773671325e is finished
2021-04-13 11:29:53,757 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e78fcde0-29ce-4e0d-8d26-bb2bfb84c659, queue=awx_private_queue)
[2021-04-13 11:29:53,757: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e78fcde0-29ce-4e0d-8d26-bb2bfb84c659, queue=awx_private_queue)
2021-04-13 11:29:53,817 DEBUG awx.main.dispatch delivered e78fcde0-29ce-4e0d-8d26-bb2bfb84c659 to worker[202] qsize 0
2021-04-13 11:29:53,825 DEBUG awx.main.dispatch task e78fcde0-29ce-4e0d-8d26-bb2bfb84c659 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:29:53,842 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:29:53,844 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:29:53,849 DEBUG awx.main.dispatch task e78fcde0-29ce-4e0d-8d26-bb2bfb84c659 is finished
RESULT 2
OKREADY
2021-04-13 11:30:04,094 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(0f8042eb-2ba5-4cd0-97cd-1ad9503c3a76, queue=dev-awx-0)
[2021-04-13 11:30:04,094: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(0f8042eb-2ba5-4cd0-97cd-1ad9503c3a76, queue=dev-awx-0)
2021-04-13 11:30:04,123 DEBUG awx.main.dispatch delivered 0f8042eb-2ba5-4cd0-97cd-1ad9503c3a76 to worker[216] qsize 0
2021-04-13 11:30:04,134 DEBUG awx.main.dispatch task 0f8042eb-2ba5-4cd0-97cd-1ad9503c3a76 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:30:04,161 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140082896053824_5" does not exist
2021-04-13 11:30:04,167 DEBUG awx.main.dispatch task 0f8042eb-2ba5-4cd0-97cd-1ad9503c3a76 is finished
2021-04-13 11:30:05,044 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d146fd8a-0352-48be-af8c-cc6c0c443ed0, queue=awx_private_queue)
[2021-04-13 11:30:05,044: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d146fd8a-0352-48be-af8c-cc6c0c443ed0, queue=awx_private_queue)
2021-04-13 11:30:05,073 DEBUG awx.main.dispatch delivered d146fd8a-0352-48be-af8c-cc6c0c443ed0 to worker[216] qsize 0
2021-04-13 11:30:05,086 DEBUG awx.main.dispatch task d146fd8a-0352-48be-af8c-cc6c0c443ed0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:30:05,099 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:30:05,102 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:30:05,109 DEBUG awx.main.dispatch task d146fd8a-0352-48be-af8c-cc6c0c443ed0 is finished
2021-04-13 11:30:23,683 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(3955b01f-4dce-4c5b-9d61-9a16d631060d, queue=dev-awx-0)
[2021-04-13 11:30:23,683: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(3955b01f-4dce-4c5b-9d61-9a16d631060d, queue=dev-awx-0)
2021-04-13 11:30:23,717 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-13 11:30:23,755 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(73cd903e-5e05-4301-bb07-e89bc6065e75, queue=awx_private_queue)
[2021-04-13 11:30:23,755: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(73cd903e-5e05-4301-bb07-e89bc6065e75, queue=awx_private_queue)
2021-04-13 11:30:23,779 DEBUG awx.main.dispatch delivered 73cd903e-5e05-4301-bb07-e89bc6065e75 to worker[216] qsize 0
2021-04-13 11:30:23,788 DEBUG awx.main.dispatch task 73cd903e-5e05-4301-bb07-e89bc6065e75 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:30:23,803 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:30:23,806 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:30:23,810 DEBUG awx.main.dispatch task 73cd903e-5e05-4301-bb07-e89bc6065e75 is finished
2021-04-13 11:30:25,044 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d229a47f-4851-44a6-8201-eb9d5c308662, queue=awx_private_queue)
[2021-04-13 11:30:25,044: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d229a47f-4851-44a6-8201-eb9d5c308662, queue=awx_private_queue)
2021-04-13 11:30:25,075 DEBUG awx.main.dispatch delivered d229a47f-4851-44a6-8201-eb9d5c308662 to worker[202] qsize 0
2021-04-13 11:30:25,083 DEBUG awx.main.dispatch task d229a47f-4851-44a6-8201-eb9d5c308662 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:30:25,101 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:30:25,102 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:30:25,107 DEBUG awx.main.dispatch task d229a47f-4851-44a6-8201-eb9d5c308662 is finished
2021-04-13 11:30:45,054 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3f1f8d8b-df31-4384-8568-876f38a15aec, queue=awx_private_queue)
[2021-04-13 11:30:45,054: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3f1f8d8b-df31-4384-8568-876f38a15aec, queue=awx_private_queue)
2021-04-13 11:30:45,087 DEBUG awx.main.dispatch delivered 3f1f8d8b-df31-4384-8568-876f38a15aec to worker[216] qsize 0
2021-04-13 11:30:45,098 DEBUG awx.main.dispatch task 3f1f8d8b-df31-4384-8568-876f38a15aec starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:30:45,112 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:30:45,113 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:30:45,123 DEBUG awx.main.dispatch task 3f1f8d8b-df31-4384-8568-876f38a15aec is finished
2021-04-13 11:30:53,756 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8beb863d-10a9-4d63-ab07-90bd4f619ed7, queue=awx_private_queue)
[2021-04-13 11:30:53,756: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8beb863d-10a9-4d63-ab07-90bd4f619ed7, queue=awx_private_queue)
2021-04-13 11:30:53,784 DEBUG awx.main.dispatch delivered 8beb863d-10a9-4d63-ab07-90bd4f619ed7 to worker[204] qsize 0
2021-04-13 11:30:53,797 DEBUG awx.main.dispatch task 8beb863d-10a9-4d63-ab07-90bd4f619ed7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:30:53,821 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:30:53,824 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:30:53,828 DEBUG awx.main.dispatch task 8beb863d-10a9-4d63-ab07-90bd4f619ed7 is finished
RESULT 2
OKREADY
2021-04-13 11:31:04,100 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f8ddce10-c472-4240-aa57-f194813f9d70, queue=dev-awx-0)
[2021-04-13 11:31:04,100: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f8ddce10-c472-4240-aa57-f194813f9d70, queue=dev-awx-0)
2021-04-13 11:31:04,136 DEBUG awx.main.dispatch delivered f8ddce10-c472-4240-aa57-f194813f9d70 to worker[202] qsize 0
2021-04-13 11:31:04,146 DEBUG awx.main.dispatch task f8ddce10-c472-4240-aa57-f194813f9d70 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:31:04,165 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140082896053824_3" does not exist
2021-04-13 11:31:04,171 DEBUG awx.main.dispatch task f8ddce10-c472-4240-aa57-f194813f9d70 is finished
2021-04-13 11:31:05,066 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a2e8d422-d0f6-4b83-ac2b-617d4c255e80, queue=awx_private_queue)
[2021-04-13 11:31:05,066: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a2e8d422-d0f6-4b83-ac2b-617d4c255e80, queue=awx_private_queue)
2021-04-13 11:31:05,093 DEBUG awx.main.dispatch delivered a2e8d422-d0f6-4b83-ac2b-617d4c255e80 to worker[204] qsize 0
2021-04-13 11:31:05,105 DEBUG awx.main.dispatch task a2e8d422-d0f6-4b83-ac2b-617d4c255e80 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:31:05,119 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:31:05,121 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:31:05,126 DEBUG awx.main.dispatch task a2e8d422-d0f6-4b83-ac2b-617d4c255e80 is finished
2021-04-13 11:31:23,692 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(71f15afc-7d16-4c25-87ca-67320110b3fe, queue=dev-awx-0)
[2021-04-13 11:31:23,692: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(71f15afc-7d16-4c25-87ca-67320110b3fe, queue=dev-awx-0)
2021-04-13 11:31:23,725 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-13 11:31:23,757 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(fad2690c-8e06-4f58-b2d1-407be835cae2, queue=awx_private_queue)
[2021-04-13 11:31:23,757: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(fad2690c-8e06-4f58-b2d1-407be835cae2, queue=awx_private_queue)
2021-04-13 11:31:23,794 DEBUG awx.main.dispatch delivered fad2690c-8e06-4f58-b2d1-407be835cae2 to worker[209] qsize 0
2021-04-13 11:31:23,803 DEBUG awx.main.dispatch task fad2690c-8e06-4f58-b2d1-407be835cae2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:31:23,827 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:31:23,830 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:31:23,834 DEBUG awx.main.dispatch task fad2690c-8e06-4f58-b2d1-407be835cae2 is finished
2021-04-13 11:31:25,061 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(37071013-285a-4aa1-a51b-fdaaec32a6b7, queue=awx_private_queue)
[2021-04-13 11:31:25,061: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(37071013-285a-4aa1-a51b-fdaaec32a6b7, queue=awx_private_queue)
2021-04-13 11:31:25,090 DEBUG awx.main.dispatch delivered 37071013-285a-4aa1-a51b-fdaaec32a6b7 to worker[204] qsize 0
2021-04-13 11:31:25,098 DEBUG awx.main.dispatch task 37071013-285a-4aa1-a51b-fdaaec32a6b7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:31:25,108 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:31:25,109 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:31:25,113 DEBUG awx.main.dispatch task 37071013-285a-4aa1-a51b-fdaaec32a6b7 is finished
2021-04-13 11:31:45,062 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(030a7da9-575c-4ba5-97e1-6d1d1ed3e1cd, queue=awx_private_queue)
[2021-04-13 11:31:45,062: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(030a7da9-575c-4ba5-97e1-6d1d1ed3e1cd, queue=awx_private_queue)
2021-04-13 11:31:45,101 DEBUG awx.main.dispatch delivered 030a7da9-575c-4ba5-97e1-6d1d1ed3e1cd to worker[216] qsize 0
2021-04-13 11:31:45,114 DEBUG awx.main.dispatch task 030a7da9-575c-4ba5-97e1-6d1d1ed3e1cd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:31:45,138 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:31:45,140 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:31:45,145 DEBUG awx.main.dispatch task 030a7da9-575c-4ba5-97e1-6d1d1ed3e1cd is finished
2021-04-13 11:31:49,841 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(92d5ddf7-4db3-4cf6-972f-4d89f0829774, queue=dev-awx-0)
[2021-04-13 11:31:49,841: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(92d5ddf7-4db3-4cf6-972f-4d89f0829774, queue=dev-awx-0)
2021-04-13 11:31:49,865 DEBUG awx.main.dispatch delivered 92d5ddf7-4db3-4cf6-972f-4d89f0829774 to worker[202] qsize 0
2021-04-13 11:31:49,876 DEBUG awx.main.dispatch task 92d5ddf7-4db3-4cf6-972f-4d89f0829774 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-04-13 11:31:49,892 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-04-13 11:31:49,911 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 470, in awx_isolated_heartbeat
for isolated_instance in isolated_instance_qs:
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-04-13 11:31:49,916 DEBUG awx.main.dispatch task 92d5ddf7-4db3-4cf6-972f-4d89f0829774 is finished
2021-04-13 11:31:53,756 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e77e2150-75ba-4294-9344-f251a45f2c28, queue=awx_private_queue)
[2021-04-13 11:31:53,756: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e77e2150-75ba-4294-9344-f251a45f2c28, queue=awx_private_queue)
2021-04-13 11:31:53,796 DEBUG awx.main.dispatch delivered e77e2150-75ba-4294-9344-f251a45f2c28 to worker[216] qsize 0
2021-04-13 11:31:53,804 DEBUG awx.main.dispatch task e77e2150-75ba-4294-9344-f251a45f2c28 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:31:53,820 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:31:53,833 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:31:53,841 DEBUG awx.main.dispatch task e77e2150-75ba-4294-9344-f251a45f2c28 is finished
RESULT 2
OKREADY
2021-04-13 11:32:04,097 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b6fb9de7-814c-4c3a-ad6c-ddce95af3cdc, queue=dev-awx-0)
[2021-04-13 11:32:04,097: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b6fb9de7-814c-4c3a-ad6c-ddce95af3cdc, queue=dev-awx-0)
2021-04-13 11:32:04,123 DEBUG awx.main.dispatch delivered b6fb9de7-814c-4c3a-ad6c-ddce95af3cdc to worker[202] qsize 0
2021-04-13 11:32:04,132 DEBUG awx.main.dispatch task b6fb9de7-814c-4c3a-ad6c-ddce95af3cdc starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-13 11:32:04,148 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140082896053824_4" does not exist
2021-04-13 11:32:04,152 DEBUG awx.main.dispatch task b6fb9de7-814c-4c3a-ad6c-ddce95af3cdc is finished
2021-04-13 11:32:05,063 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cd63e33b-39b1-4120-a69d-27c9632870c0, queue=awx_private_queue)
[2021-04-13 11:32:05,063: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cd63e33b-39b1-4120-a69d-27c9632870c0, queue=awx_private_queue)
2021-04-13 11:32:05,099 DEBUG awx.main.dispatch delivered cd63e33b-39b1-4120-a69d-27c9632870c0 to worker[202] qsize 0
2021-04-13 11:32:05,111 DEBUG awx.main.dispatch task cd63e33b-39b1-4120-a69d-27c9632870c0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:32:05,123 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:32:05,125 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:32:05,129 DEBUG awx.main.dispatch task cd63e33b-39b1-4120-a69d-27c9632870c0 is finished
2021-04-13 11:32:23,703 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(af5402d4-6c7b-4188-b271-fe6b4d97008a, queue=dev-awx-0)
[2021-04-13 11:32:23,703: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(af5402d4-6c7b-4188-b271-fe6b4d97008a, queue=dev-awx-0)
2021-04-13 11:32:23,741 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-13 11:32:23,756 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(81d18e17-d825-4693-b658-d1a071f5205e, queue=awx_private_queue)
[2021-04-13 11:32:23,756: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(81d18e17-d825-4693-b658-d1a071f5205e, queue=awx_private_queue)
2021-04-13 11:32:23,780 DEBUG awx.main.dispatch delivered 81d18e17-d825-4693-b658-d1a071f5205e to worker[216] qsize 0
2021-04-13 11:32:23,790 DEBUG awx.main.dispatch task 81d18e17-d825-4693-b658-d1a071f5205e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-13 11:32:23,813 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-13 11:32:23,816 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-13 11:32:23,822 DEBUG awx.main.dispatch task 81d18e17-d825-4693-b658-d1a071f5205e is finished
2021-04-13 11:32:25,063 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(78a0197c-4111-4403-b9ef-c5440158eb7a, queue=awx_private_queue)
[2021-04-13 11:32:25,063: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(78a0197c-4111-4403-b9ef-c5440158eb7a, queue=awx_private_queue)
2021-04-13 11:32:25,096 DEBUG awx.main.dispatch delivered 78a0197c-4111-4403-b9ef-c5440158eb7a to worker[216] qsize 0
2021-04-13 11:32:25,103 DEBUG awx.main.dispatch task 78a0197c-4111-4403-b9ef-c5440158eb7a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:32:25,115 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:32:25,116 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-13 11:32:25,121 DEBUG awx.main.dispatch task 78a0197c-4111-4403-b9ef-c5440158eb7a is finished
2021-04-13 11:32:45,098 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1a4e7260-7953-46ba-8e0a-bc9c416f36b8, queue=awx_private_queue)
[2021-04-13 11:32:45,098: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1a4e7260-7953-46ba-8e0a-bc9c416f36b8, queue=awx_private_queue)
2021-04-13 11:32:45,132 DEBUG awx.main.dispatch delivered 1a4e7260-7953-46ba-8e0a-bc9c416f36b8 to worker[204] qsize 0
2021-04-13 11:32:45,147 DEBUG awx.main.dispatch task 1a4e7260-7953-46ba-8e0a-bc9c416f36b8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-13 11:32:45,176 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-13 11:32:45,177 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return sel