By type
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"elapsed": 0,
"match_groupdict": {},
"match_groups": [],
"path": null,
"port": 5432,
"search_regex": null,
"state": "started"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"db": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
}
Default organization added.
Demo Credential, Inventory, and Job Template added.
(changed: True)
2021-04-07 07:55:31,522 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(aab71fa1-2f3d-4441-8b9a-45035bd4d80b, queue=awx_private_queue)
Successfully registered instance dev-awx-0
(changed: True)
2021-04-07 07:55:48,188 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(52d89fc0-366d-44e5-8eab-8d9fe2882675, queue=awx_private_queue)
2021-04-07 07:55:48,231 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(8a7c724d-6339-4cb5-9392-e7be0b5bcb0a, queue=awx_private_queue)
Creating instance group tower
(changed: True)
2021-04-07 07:55:49,128 INFO RPC interface 'supervisor' initialized
2021-04-07 07:55:49,128 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2021-04-07 07:55:49,128 INFO supervisord started with pid 150
2021-04-07 07:55:50,135 INFO spawned: 'awx-config-watcher' with pid 153
2021-04-07 07:55:50,139 INFO spawned: 'dispatcher' with pid 154
2021-04-07 07:55:50,143 INFO spawned: 'callback-receiver' with pid 155
2021-04-07 07:55:50,148 INFO spawned: 'channels-worker' with pid 156
READY
2021-04-07 07:55:51,291 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-07 07:55:51,291 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-07 07:55:51,291 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-07 07:55:51,291 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-07 07:55:53,520 - INFO - runworker - Using single-threaded worker.
2021-04-07 07:55:53,521 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer)
2021-04-07 07:55:53,522 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive
2021-04-07 07:55:53,602 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:175
2021-04-07 07:55:53,602 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:175
2021-04-07 07:55:53,608 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-04-07 07:55:53,608 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-04-07 07:55:53,615 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2021-04-07 07:55:53,615 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2021-04-07 07:55:53,621 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2021-04-07 07:55:53,621 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2021-04-07 07:55:53,797 WARNING awx.main.dispatch scaling up worker pid:180
2021-04-07 07:55:53,808 WARNING awx.main.dispatch scaling up worker pid:181
2021-04-07 07:55:53,814 ERROR celery.beat Removing corrupted schedule file '/var/lib/awx/beat.db': error(11, 'Resource temporarily unavailable')
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 485, in setup_schedule
self._store = self._open_schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 475, in _open_schedule
return self.persistence.open(self.schedule_filename, writeback=True)
File "/usr/lib64/python3.6/shelve.py", line 243, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/usr/lib64/python3.6/shelve.py", line 227, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
File "/usr/lib64/python3.6/dbm/__init__.py", line 94, in open
return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable
2021-04-07 07:55:53,818 WARNING awx.main.dispatch scaling up worker pid:182
2021-04-07 07:55:53,827 WARNING awx.main.dispatch scaling up worker pid:183
2021-04-07 07:55:53,831 DEBUG awx.main.tasks Syncing Schedules
2021-04-07 07:55:54,128 DEBUG awx.main.tasks Waited 0.001556396484375 seconds to obtain lock name: cluster_policy_lock
2021-04-07 07:55:54,164 DEBUG awx.main.tasks Total non-isolated instances:1 available for policy: 1
2021-04-07 07:55:54,164 DEBUG awx.main.tasks Policy percentage, adding Instances [1] to Group tower
2021-04-07 07:55:54,164 DEBUG awx.main.tasks Adding instances [1] to group tower
2021-04-07 07:55:54,193 DEBUG awx.main.tasks Cluster policy computation finished in 0.06469392776489258 seconds
2021-04-07 07:55:54,199 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 07:55:54,281 DEBUG awx.main.dispatch listening on ['awx_private_queue [direct]', 'dev-awx-0 [direct]', 'tower_broadcast_all_dev-awx-0 [fanout]']
RESULT 2
OKREADY
2021-04-07 07:56:13,972 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f7c74bc3-e15e-4355-ac36-4d8adf4e1a28, queue=awx_private_queue)
[2021-04-07 07:56:13,972: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f7c74bc3-e15e-4355-ac36-4d8adf4e1a28, queue=awx_private_queue)
2021-04-07 07:56:13,997 DEBUG awx.main.dispatch delivered f7c74bc3-e15e-4355-ac36-4d8adf4e1a28 to worker[182] qsize 0
2021-04-07 07:56:14,003 DEBUG awx.main.dispatch task f7c74bc3-e15e-4355-ac36-4d8adf4e1a28 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 07:56:14,008 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 07:56:14,119 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 07:56:14,502 DEBUG awx.main.dispatch task f7c74bc3-e15e-4355-ac36-4d8adf4e1a28 is finished
2021-04-07 07:56:23,938 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5386f82f-6131-43fe-9868-772839cba92e, queue=awx_private_queue)
[2021-04-07 07:56:23,938: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5386f82f-6131-43fe-9868-772839cba92e, queue=awx_private_queue)
2021-04-07 07:56:23,961 DEBUG awx.main.dispatch delivered 5386f82f-6131-43fe-9868-772839cba92e to worker[182] qsize 0
2021-04-07 07:56:23,968 DEBUG awx.main.dispatch task 5386f82f-6131-43fe-9868-772839cba92e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 07:56:23,995 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 07:56:24,094 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 07:56:24.011554+00:00
2021-04-07 07:56:24,301 DEBUG awx.main.dispatch task 5386f82f-6131-43fe-9868-772839cba92e is finished
2021-04-07 07:56:33,952 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d76756b6-901c-4b04-958e-45c06e9b7d39, queue=awx_private_queue)
[2021-04-07 07:56:33,952: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d76756b6-901c-4b04-958e-45c06e9b7d39, queue=awx_private_queue)
2021-04-07 07:56:33,981 DEBUG awx.main.dispatch delivered d76756b6-901c-4b04-958e-45c06e9b7d39 to worker[180] qsize 0
2021-04-07 07:56:33,997 DEBUG awx.main.dispatch task d76756b6-901c-4b04-958e-45c06e9b7d39 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 07:56:34,005 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 07:56:34,043 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 07:56:34,099 DEBUG awx.main.dispatch task d76756b6-901c-4b04-958e-45c06e9b7d39 is finished
2021-04-07 07:56:53,946 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6f202369-f51a-4a07-85a3-f4776e67d3c5, queue=dev-awx-0)
[2021-04-07 07:56:53,946: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6f202369-f51a-4a07-85a3-f4776e67d3c5, queue=dev-awx-0)
2021-04-07 07:56:54,191 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(195aead1-d0c1-4c97-a48f-b14762faee24, queue=dev-awx-0)
[2021-04-07 07:56:54,191: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(195aead1-d0c1-4c97-a48f-b14762faee24, queue=dev-awx-0)
2021-04-07 07:56:54,204 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8d0ab2de-751c-4836-ab79-2f1458f3318b, queue=awx_private_queue)
[2021-04-07 07:56:54,204: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8d0ab2de-751c-4836-ab79-2f1458f3318b, queue=awx_private_queue)
2021-04-07 07:56:54,217 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c1b1e773-91dd-429f-910f-0a11553e1a98, queue=awx_private_queue)
[2021-04-07 07:56:54,217: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c1b1e773-91dd-429f-910f-0a11553e1a98, queue=awx_private_queue)
2021-04-07 07:56:54,241 DEBUG awx.main.dispatch delivered 6f202369-f51a-4a07-85a3-f4776e67d3c5 to worker[180] qsize 0
2021-04-07 07:56:54,244 DEBUG awx.main.dispatch delivered 195aead1-d0c1-4c97-a48f-b14762faee24 to worker[183] qsize 0
2021-04-07 07:56:54,246 DEBUG awx.main.dispatch delivered 8d0ab2de-751c-4836-ab79-2f1458f3318b to worker[182] qsize 0
2021-04-07 07:56:54,244 DEBUG awx.main.dispatch task 6f202369-f51a-4a07-85a3-f4776e67d3c5 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 07:56:54,248 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 07:56:54,247 DEBUG awx.main.dispatch delivered c1b1e773-91dd-429f-910f-0a11553e1a98 to worker[181] qsize 0
2021-04-07 07:56:54,249 DEBUG awx.main.dispatch task 8d0ab2de-751c-4836-ab79-2f1458f3318b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 07:56:54,248 DEBUG awx.main.dispatch task 195aead1-d0c1-4c97-a48f-b14762faee24 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 07:56:54,254 DEBUG awx.main.dispatch task c1b1e773-91dd-429f-910f-0a11553e1a98 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 07:56:54,260 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 07:56:54,269 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 07:56:54,273 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 07:56:23.995616+00:00
2021-04-07 07:56:54,281 DEBUG awx.main.dispatch task 195aead1-d0c1-4c97-a48f-b14762faee24 is finished
2021-04-07 07:56:54,286 DEBUG awx.main.dispatch task 6f202369-f51a-4a07-85a3-f4776e67d3c5 is finished
2021-04-07 07:56:54,288 DEBUG awx.main.dispatch task 8d0ab2de-751c-4836-ab79-2f1458f3318b is finished
2021-04-07 07:56:54,292 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 07:56:54,338 DEBUG awx.main.dispatch task c1b1e773-91dd-429f-910f-0a11553e1a98 is finished
RESULT 2
OKREADY
2021-04-07 07:57:14,240 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(761d03cb-08d1-46f8-89a5-0ee8c7053ef3, queue=awx_private_queue)
[2021-04-07 07:57:14,240: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(761d03cb-08d1-46f8-89a5-0ee8c7053ef3, queue=awx_private_queue)
2021-04-07 07:57:14,280 DEBUG awx.main.dispatch delivered 761d03cb-08d1-46f8-89a5-0ee8c7053ef3 to worker[181] qsize 0
2021-04-07 07:57:14,289 DEBUG awx.main.dispatch task 761d03cb-08d1-46f8-89a5-0ee8c7053ef3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 07:57:14,295 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 07:57:14,326 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 07:57:14,372 DEBUG awx.main.dispatch task 761d03cb-08d1-46f8-89a5-0ee8c7053ef3 is finished
2021-04-07 07:57:24,206 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2603955f-1eea-4bd1-a08a-2a740df8fd9e, queue=awx_private_queue)
[2021-04-07 07:57:24,206: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2603955f-1eea-4bd1-a08a-2a740df8fd9e, queue=awx_private_queue)
2021-04-07 07:57:24,226 DEBUG awx.main.dispatch delivered 2603955f-1eea-4bd1-a08a-2a740df8fd9e to worker[182] qsize 0
2021-04-07 07:57:24,230 DEBUG awx.main.dispatch task 2603955f-1eea-4bd1-a08a-2a740df8fd9e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 07:57:24,248 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 07:57:24,254 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 07:56:54.269634+00:00
2021-04-07 07:57:24,271 DEBUG awx.main.dispatch task 2603955f-1eea-4bd1-a08a-2a740df8fd9e is finished
2021-04-07 07:57:34,233 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1aa6fe10-b2a9-4a53-9190-742109eeb20a, queue=awx_private_queue)
[2021-04-07 07:57:34,233: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1aa6fe10-b2a9-4a53-9190-742109eeb20a, queue=awx_private_queue)
2021-04-07 07:57:34,254 DEBUG awx.main.dispatch delivered 1aa6fe10-b2a9-4a53-9190-742109eeb20a to worker[182] qsize 0
2021-04-07 07:57:34,258 DEBUG awx.main.dispatch task 1aa6fe10-b2a9-4a53-9190-742109eeb20a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 07:57:34,261 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 07:57:34,283 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 07:57:34,323 DEBUG awx.main.dispatch task 1aa6fe10-b2a9-4a53-9190-742109eeb20a is finished
2021-04-07 07:57:53,954 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8ee240b9-513e-4437-a937-0f36bcef5561, queue=dev-awx-0)
[2021-04-07 07:57:53,954: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8ee240b9-513e-4437-a937-0f36bcef5561, queue=dev-awx-0)
2021-04-07 07:57:54,192 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c17a9bde-33f0-4f50-8349-0400071d7e3f, queue=dev-awx-0)
[2021-04-07 07:57:54,192: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c17a9bde-33f0-4f50-8349-0400071d7e3f, queue=dev-awx-0)
2021-04-07 07:57:54,205 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(70e415e2-d322-4871-ab65-a40cf212088b, queue=awx_private_queue)
[2021-04-07 07:57:54,205: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(70e415e2-d322-4871-ab65-a40cf212088b, queue=awx_private_queue)
2021-04-07 07:57:54,217 DEBUG awx.main.dispatch delivered 8ee240b9-513e-4437-a937-0f36bcef5561 to worker[181] qsize 0
2021-04-07 07:57:54,224 DEBUG awx.main.dispatch delivered c17a9bde-33f0-4f50-8349-0400071d7e3f to worker[182] qsize 0
2021-04-07 07:57:54,226 DEBUG awx.main.dispatch delivered 70e415e2-d322-4871-ab65-a40cf212088b to worker[183] qsize 0
2021-04-07 07:57:54,226 DEBUG awx.main.dispatch task 8ee240b9-513e-4437-a937-0f36bcef5561 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 07:57:54,228 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 07:57:54,227 DEBUG awx.main.dispatch task c17a9bde-33f0-4f50-8349-0400071d7e3f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 07:57:54,232 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1be4554c-e187-4a14-8739-4c9f8cf07b77, queue=awx_private_queue)
[2021-04-07 07:57:54,232: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1be4554c-e187-4a14-8739-4c9f8cf07b77, queue=awx_private_queue)
2021-04-07 07:57:54,231 DEBUG awx.main.dispatch task 70e415e2-d322-4871-ab65-a40cf212088b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 07:57:54,249 DEBUG awx.main.dispatch delivered 1be4554c-e187-4a14-8739-4c9f8cf07b77 to worker[180] qsize 0
2021-04-07 07:57:54,253 DEBUG awx.main.dispatch task 1be4554c-e187-4a14-8739-4c9f8cf07b77 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 07:57:54,257 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 07:57:54,266 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 07:57:54,269 DEBUG awx.main.dispatch task c17a9bde-33f0-4f50-8349-0400071d7e3f is finished
2021-04-07 07:57:54,281 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 07:57:54,275 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 07:57:24.248799+00:00
2021-04-07 07:57:54,280 DEBUG awx.main.dispatch task 8ee240b9-513e-4437-a937-0f36bcef5561 is finished
2021-04-07 07:57:54,302 DEBUG awx.main.dispatch task 70e415e2-d322-4871-ab65-a40cf212088b is finished
2021-04-07 07:57:54,326 DEBUG awx.main.dispatch task 1be4554c-e187-4a14-8739-4c9f8cf07b77 is finished
RESULT 2
OKREADY
2021-04-07 07:58:14,253 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(226d074b-975a-47a7-bd02-e963b99e0f68, queue=awx_private_queue)
[2021-04-07 07:58:14,253: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(226d074b-975a-47a7-bd02-e963b99e0f68, queue=awx_private_queue)
2021-04-07 07:58:14,298 DEBUG awx.main.dispatch delivered 226d074b-975a-47a7-bd02-e963b99e0f68 to worker[182] qsize 0
2021-04-07 07:58:14,308 DEBUG awx.main.dispatch task 226d074b-975a-47a7-bd02-e963b99e0f68 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 07:58:14,315 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 07:58:14,340 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 07:58:14,373 DEBUG awx.main.dispatch task 226d074b-975a-47a7-bd02-e963b99e0f68 is finished
2021-04-07 07:58:24,207 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ffb62e23-7b7c-4c46-a92e-3c60f10273e1, queue=awx_private_queue)
[2021-04-07 07:58:24,207: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ffb62e23-7b7c-4c46-a92e-3c60f10273e1, queue=awx_private_queue)
2021-04-07 07:58:24,229 DEBUG awx.main.dispatch delivered ffb62e23-7b7c-4c46-a92e-3c60f10273e1 to worker[182] qsize 0
2021-04-07 07:58:24,233 DEBUG awx.main.dispatch task ffb62e23-7b7c-4c46-a92e-3c60f10273e1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 07:58:24,247 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 07:58:24,251 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 07:57:54.266574+00:00
2021-04-07 07:58:24,271 DEBUG awx.main.dispatch task ffb62e23-7b7c-4c46-a92e-3c60f10273e1 is finished
2021-04-07 07:58:34,248 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5b71629a-bd47-47d0-9fb7-d22b08039210, queue=awx_private_queue)
[2021-04-07 07:58:34,248: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5b71629a-bd47-47d0-9fb7-d22b08039210, queue=awx_private_queue)
2021-04-07 07:58:34,274 DEBUG awx.main.dispatch delivered 5b71629a-bd47-47d0-9fb7-d22b08039210 to worker[181] qsize 0
2021-04-07 07:58:34,280 DEBUG awx.main.dispatch task 5b71629a-bd47-47d0-9fb7-d22b08039210 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 07:58:34,284 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 07:58:34,309 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 07:58:34,352 DEBUG awx.main.dispatch task 5b71629a-bd47-47d0-9fb7-d22b08039210 is finished
2021-04-07 07:58:53,961 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(90f51ff9-6a43-47e1-90df-3b95d62c6c84, queue=dev-awx-0)
[2021-04-07 07:58:53,961: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(90f51ff9-6a43-47e1-90df-3b95d62c6c84, queue=dev-awx-0)
2021-04-07 07:58:54,189 DEBUG awx.main.dispatch delivered 90f51ff9-6a43-47e1-90df-3b95d62c6c84 to worker[183] qsize 0
2021-04-07 07:58:54,192 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(dafa4383-02d6-44a5-967b-057ca88f8464, queue=dev-awx-0)
[2021-04-07 07:58:54,192: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(dafa4383-02d6-44a5-967b-057ca88f8464, queue=dev-awx-0)
2021-04-07 07:58:54,195 DEBUG awx.main.dispatch task 90f51ff9-6a43-47e1-90df-3b95d62c6c84 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 07:58:54,206 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 07:58:54,215 DEBUG awx.main.dispatch delivered dafa4383-02d6-44a5-967b-057ca88f8464 to worker[181] qsize 0
2021-04-07 07:58:54,216 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d2d298fd-db7e-4456-8589-2bf0a2eb09ed, queue=awx_private_queue)
[2021-04-07 07:58:54,216: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d2d298fd-db7e-4456-8589-2bf0a2eb09ed, queue=awx_private_queue)
2021-04-07 07:58:54,217 DEBUG awx.main.dispatch task dafa4383-02d6-44a5-967b-057ca88f8464 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 07:58:54,233 DEBUG awx.main.dispatch delivered d2d298fd-db7e-4456-8589-2bf0a2eb09ed to worker[182] qsize 0
2021-04-07 07:58:54,236 DEBUG awx.main.dispatch task dafa4383-02d6-44a5-967b-057ca88f8464 is finished
2021-04-07 07:58:54,235 DEBUG awx.main.dispatch task d2d298fd-db7e-4456-8589-2bf0a2eb09ed starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 07:58:54,241 DEBUG awx.main.dispatch task 90f51ff9-6a43-47e1-90df-3b95d62c6c84 is finished
2021-04-07 07:58:54,245 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 07:58:54,246 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c48bf3c8-c57f-4076-821a-497e89ca9ffb, queue=awx_private_queue)
[2021-04-07 07:58:54,246: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c48bf3c8-c57f-4076-821a-497e89ca9ffb, queue=awx_private_queue)
2021-04-07 07:58:54,249 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 07:58:24.247713+00:00
2021-04-07 07:58:54,264 DEBUG awx.main.dispatch delivered c48bf3c8-c57f-4076-821a-497e89ca9ffb to worker[183] qsize 0
2021-04-07 07:58:54,269 DEBUG awx.main.dispatch task d2d298fd-db7e-4456-8589-2bf0a2eb09ed is finished
2021-04-07 07:58:54,271 DEBUG awx.main.dispatch task c48bf3c8-c57f-4076-821a-497e89ca9ffb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 07:58:54,273 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 07:58:54,292 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 07:58:54,341 DEBUG awx.main.dispatch task c48bf3c8-c57f-4076-821a-497e89ca9ffb is finished
RESULT 2
OKREADY
2021-04-07 07:59:14,260 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4f386d95-1f0f-455f-a3fc-7a65864c1da4, queue=awx_private_queue)
[2021-04-07 07:59:14,260: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4f386d95-1f0f-455f-a3fc-7a65864c1da4, queue=awx_private_queue)
2021-04-07 07:59:14,295 DEBUG awx.main.dispatch delivered 4f386d95-1f0f-455f-a3fc-7a65864c1da4 to worker[180] qsize 0
2021-04-07 07:59:14,304 DEBUG awx.main.dispatch task 4f386d95-1f0f-455f-a3fc-7a65864c1da4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 07:59:14,312 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 07:59:14,339 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 07:59:14,373 DEBUG awx.main.dispatch task 4f386d95-1f0f-455f-a3fc-7a65864c1da4 is finished
2021-04-07 07:59:24,222 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2b6d5cfb-0702-4f22-9544-89e3d464086b, queue=awx_private_queue)
[2021-04-07 07:59:24,222: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2b6d5cfb-0702-4f22-9544-89e3d464086b, queue=awx_private_queue)
2021-04-07 07:59:24,249 DEBUG awx.main.dispatch delivered 2b6d5cfb-0702-4f22-9544-89e3d464086b to worker[180] qsize 0
2021-04-07 07:59:24,256 DEBUG awx.main.dispatch task 2b6d5cfb-0702-4f22-9544-89e3d464086b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 07:59:24,270 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 07:59:24,276 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 07:58:54.246020+00:00
2021-04-07 07:59:24,306 DEBUG awx.main.dispatch task 2b6d5cfb-0702-4f22-9544-89e3d464086b is finished
2021-04-07 07:59:34,254 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(83f76ae8-f18d-4b6b-9748-a19df8f8e9ad, queue=awx_private_queue)
[2021-04-07 07:59:34,254: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(83f76ae8-f18d-4b6b-9748-a19df8f8e9ad, queue=awx_private_queue)
2021-04-07 07:59:34,275 DEBUG awx.main.dispatch delivered 83f76ae8-f18d-4b6b-9748-a19df8f8e9ad to worker[182] qsize 0
2021-04-07 07:59:34,280 DEBUG awx.main.dispatch task 83f76ae8-f18d-4b6b-9748-a19df8f8e9ad starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 07:59:34,285 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 07:59:34,309 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 07:59:34,345 DEBUG awx.main.dispatch task 83f76ae8-f18d-4b6b-9748-a19df8f8e9ad is finished
2021-04-07 07:59:53,971 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f572e0d1-0ddd-4a82-8ed9-67c434636e90, queue=dev-awx-0)
[2021-04-07 07:59:53,971: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f572e0d1-0ddd-4a82-8ed9-67c434636e90, queue=dev-awx-0)
2021-04-07 07:59:54,242 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(1d1c35d9-42e9-4ac7-b316-502782cbca39, queue=dev-awx-0)
[2021-04-07 07:59:54,242: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(1d1c35d9-42e9-4ac7-b316-502782cbca39, queue=dev-awx-0)
2021-04-07 07:59:54,260 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ba51fa87-84a8-4ead-953c-36f7c64d1d9b, queue=awx_private_queue)
[2021-04-07 07:59:54,260: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ba51fa87-84a8-4ead-953c-36f7c64d1d9b, queue=awx_private_queue)
2021-04-07 07:59:54,273 DEBUG awx.main.dispatch delivered f572e0d1-0ddd-4a82-8ed9-67c434636e90 to worker[180] qsize 0
2021-04-07 07:59:54,278 DEBUG awx.main.dispatch delivered 1d1c35d9-42e9-4ac7-b316-502782cbca39 to worker[183] qsize 0
2021-04-07 07:59:54,282 DEBUG awx.main.dispatch task f572e0d1-0ddd-4a82-8ed9-67c434636e90 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 07:59:54,285 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 07:59:54,284 DEBUG awx.main.dispatch task 1d1c35d9-42e9-4ac7-b316-502782cbca39 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 07:59:54,291 DEBUG awx.main.dispatch delivered ba51fa87-84a8-4ead-953c-36f7c64d1d9b to worker[181] qsize 0
2021-04-07 07:59:54,294 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(780ff591-8299-4f03-b1c6-d5ab596cdb68, queue=awx_private_queue)
[2021-04-07 07:59:54,294: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(780ff591-8299-4f03-b1c6-d5ab596cdb68, queue=awx_private_queue)
2021-04-07 07:59:54,297 DEBUG awx.main.dispatch task ba51fa87-84a8-4ead-953c-36f7c64d1d9b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 07:59:54,306 DEBUG awx.main.dispatch task 1d1c35d9-42e9-4ac7-b316-502782cbca39 is finished
2021-04-07 07:59:54,315 DEBUG awx.main.dispatch delivered 780ff591-8299-4f03-b1c6-d5ab596cdb68 to worker[183] qsize 0
2021-04-07 07:59:54,317 DEBUG awx.main.dispatch task f572e0d1-0ddd-4a82-8ed9-67c434636e90 is finished
2021-04-07 07:59:54,319 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 07:59:54,317 DEBUG awx.main.dispatch task 780ff591-8299-4f03-b1c6-d5ab596cdb68 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 07:59:54,319 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 07:59:54,324 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 07:59:24.270928+00:00
2021-04-07 07:59:54,338 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 07:59:54,342 DEBUG awx.main.dispatch task ba51fa87-84a8-4ead-953c-36f7c64d1d9b is finished
2021-04-07 07:59:54,368 DEBUG awx.main.dispatch task 780ff591-8299-4f03-b1c6-d5ab596cdb68 is finished
RESULT 2
OKREADY
2021-04-07 08:00:14,305 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6b345541-dabb-47bf-a70d-5c275330e17d, queue=awx_private_queue)
[2021-04-07 08:00:14,305: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6b345541-dabb-47bf-a70d-5c275330e17d, queue=awx_private_queue)
2021-04-07 08:00:14,330 DEBUG awx.main.dispatch delivered 6b345541-dabb-47bf-a70d-5c275330e17d to worker[182] qsize 0
2021-04-07 08:00:14,336 DEBUG awx.main.dispatch task 6b345541-dabb-47bf-a70d-5c275330e17d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:00:14,339 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:00:14,363 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:00:14,403 DEBUG awx.main.dispatch task 6b345541-dabb-47bf-a70d-5c275330e17d is finished
2021-04-07 08:00:24,261 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ed32ef54-46b5-4d9a-9046-ae45e2c0679a, queue=awx_private_queue)
[2021-04-07 08:00:24,261: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ed32ef54-46b5-4d9a-9046-ae45e2c0679a, queue=awx_private_queue)
2021-04-07 08:00:24,278 DEBUG awx.main.dispatch delivered ed32ef54-46b5-4d9a-9046-ae45e2c0679a to worker[180] qsize 0
2021-04-07 08:00:24,282 DEBUG awx.main.dispatch task ed32ef54-46b5-4d9a-9046-ae45e2c0679a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:00:24,294 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:00:24,299 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 07:59:54.319387+00:00
2021-04-07 08:00:24,324 DEBUG awx.main.dispatch task ed32ef54-46b5-4d9a-9046-ae45e2c0679a is finished
2021-04-07 08:00:34,311 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f284039f-3dc9-427b-8c17-6212036ae82c, queue=awx_private_queue)
[2021-04-07 08:00:34,311: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f284039f-3dc9-427b-8c17-6212036ae82c, queue=awx_private_queue)
2021-04-07 08:00:34,354 DEBUG awx.main.dispatch delivered f284039f-3dc9-427b-8c17-6212036ae82c to worker[181] qsize 0
2021-04-07 08:00:34,366 DEBUG awx.main.dispatch task f284039f-3dc9-427b-8c17-6212036ae82c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:00:34,373 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:00:34,394 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:00:34,421 DEBUG awx.main.dispatch task f284039f-3dc9-427b-8c17-6212036ae82c is finished
2021-04-07 08:00:53,984 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d2d99ff2-9b0b-485d-8717-f9329e25130c, queue=dev-awx-0)
[2021-04-07 08:00:53,984: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d2d99ff2-9b0b-485d-8717-f9329e25130c, queue=dev-awx-0)
2021-04-07 08:00:54,263 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(dc257ea0-c393-428e-a9ac-81923f932105, queue=dev-awx-0)
[2021-04-07 08:00:54,263: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(dc257ea0-c393-428e-a9ac-81923f932105, queue=dev-awx-0)
2021-04-07 08:00:54,280 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(00c32aeb-f6a6-4c53-858a-43c2fe3e3dea, queue=awx_private_queue)
[2021-04-07 08:00:54,280: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(00c32aeb-f6a6-4c53-858a-43c2fe3e3dea, queue=awx_private_queue)
2021-04-07 08:00:54,287 DEBUG awx.main.dispatch delivered d2d99ff2-9b0b-485d-8717-f9329e25130c to worker[182] qsize 0
2021-04-07 08:00:54,291 DEBUG awx.main.dispatch delivered dc257ea0-c393-428e-a9ac-81923f932105 to worker[183] qsize 0
2021-04-07 08:00:54,296 DEBUG awx.main.dispatch task dc257ea0-c393-428e-a9ac-81923f932105 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:00:54,293 DEBUG awx.main.dispatch task d2d99ff2-9b0b-485d-8717-f9329e25130c starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:00:54,301 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:00:54,304 DEBUG awx.main.dispatch delivered 00c32aeb-f6a6-4c53-858a-43c2fe3e3dea to worker[181] qsize 0
2021-04-07 08:00:54,305 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2ffd78df-993e-48b8-92a5-8b3aa16182c4, queue=awx_private_queue)
[2021-04-07 08:00:54,305: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2ffd78df-993e-48b8-92a5-8b3aa16182c4, queue=awx_private_queue)
2021-04-07 08:00:54,307 DEBUG awx.main.dispatch task 00c32aeb-f6a6-4c53-858a-43c2fe3e3dea starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:00:54,317 DEBUG awx.main.dispatch task dc257ea0-c393-428e-a9ac-81923f932105 is finished
2021-04-07 08:00:54,323 DEBUG awx.main.dispatch delivered 2ffd78df-993e-48b8-92a5-8b3aa16182c4 to worker[183] qsize 0
2021-04-07 08:00:54,332 DEBUG awx.main.dispatch task 2ffd78df-993e-48b8-92a5-8b3aa16182c4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:00:54,333 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:00:54,337 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:00:54,342 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:00:24.295088+00:00
2021-04-07 08:00:54,346 DEBUG awx.main.dispatch task d2d99ff2-9b0b-485d-8717-f9329e25130c is finished
2021-04-07 08:00:54,349 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:00:54,358 DEBUG awx.main.dispatch task 00c32aeb-f6a6-4c53-858a-43c2fe3e3dea is finished
2021-04-07 08:00:54,384 DEBUG awx.main.dispatch task 2ffd78df-993e-48b8-92a5-8b3aa16182c4 is finished
RESULT 2
OKREADY
2021-04-07 08:01:14,309 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(78a854f7-3c36-4c91-b35d-7d846e8bff7a, queue=awx_private_queue)
[2021-04-07 08:01:14,309: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(78a854f7-3c36-4c91-b35d-7d846e8bff7a, queue=awx_private_queue)
2021-04-07 08:01:14,338 DEBUG awx.main.dispatch delivered 78a854f7-3c36-4c91-b35d-7d846e8bff7a to worker[181] qsize 0
2021-04-07 08:01:14,341 DEBUG awx.main.dispatch task 78a854f7-3c36-4c91-b35d-7d846e8bff7a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:01:14,344 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:01:14,365 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:01:14,401 DEBUG awx.main.dispatch task 78a854f7-3c36-4c91-b35d-7d846e8bff7a is finished
2021-04-07 08:01:24,281 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(08ac6b7e-56f6-45ee-ae35-96d88a0eab87, queue=awx_private_queue)
[2021-04-07 08:01:24,281: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(08ac6b7e-56f6-45ee-ae35-96d88a0eab87, queue=awx_private_queue)
2021-04-07 08:01:24,303 DEBUG awx.main.dispatch delivered 08ac6b7e-56f6-45ee-ae35-96d88a0eab87 to worker[183] qsize 0
2021-04-07 08:01:24,306 DEBUG awx.main.dispatch task 08ac6b7e-56f6-45ee-ae35-96d88a0eab87 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:01:24,319 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:01:24,322 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:00:54.337574+00:00
2021-04-07 08:01:24,336 DEBUG awx.main.dispatch task 08ac6b7e-56f6-45ee-ae35-96d88a0eab87 is finished
2021-04-07 08:01:34,307 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(46d39984-7149-4d6f-ab2f-532c82eeca8b, queue=awx_private_queue)
[2021-04-07 08:01:34,307: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(46d39984-7149-4d6f-ab2f-532c82eeca8b, queue=awx_private_queue)
2021-04-07 08:01:34,331 DEBUG awx.main.dispatch delivered 46d39984-7149-4d6f-ab2f-532c82eeca8b to worker[182] qsize 0
2021-04-07 08:01:34,335 DEBUG awx.main.dispatch task 46d39984-7149-4d6f-ab2f-532c82eeca8b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:01:34,338 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:01:34,364 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:01:34,401 DEBUG awx.main.dispatch task 46d39984-7149-4d6f-ab2f-532c82eeca8b is finished
2021-04-07 08:01:53,991 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(50b660d5-8ada-438b-bc1f-d15c2b6b0190, queue=dev-awx-0)
[2021-04-07 08:01:53,991: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(50b660d5-8ada-438b-bc1f-d15c2b6b0190, queue=dev-awx-0)
2021-04-07 08:01:54,263 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d61fdd80-9c8a-4343-b446-b6edac4fd86f, queue=dev-awx-0)
[2021-04-07 08:01:54,263: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d61fdd80-9c8a-4343-b446-b6edac4fd86f, queue=dev-awx-0)
2021-04-07 08:01:54,275 DEBUG awx.main.dispatch delivered 50b660d5-8ada-438b-bc1f-d15c2b6b0190 to worker[181] qsize 0
2021-04-07 08:01:54,280 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(eccdf6b9-ca28-406b-9206-433f42014d69, queue=awx_private_queue)
[2021-04-07 08:01:54,280: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(eccdf6b9-ca28-406b-9206-433f42014d69, queue=awx_private_queue)
2021-04-07 08:01:54,280 DEBUG awx.main.dispatch delivered d61fdd80-9c8a-4343-b446-b6edac4fd86f to worker[180] qsize 0
2021-04-07 08:01:54,281 DEBUG awx.main.dispatch task 50b660d5-8ada-438b-bc1f-d15c2b6b0190 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:01:54,285 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:01:54,285 DEBUG awx.main.dispatch task d61fdd80-9c8a-4343-b446-b6edac4fd86f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:01:54,296 DEBUG awx.main.dispatch delivered eccdf6b9-ca28-406b-9206-433f42014d69 to worker[182] qsize 0
2021-04-07 08:01:54,300 DEBUG awx.main.dispatch task eccdf6b9-ca28-406b-9206-433f42014d69 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:01:54,306 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3760819c-2171-473f-8156-3191f33544b1, queue=awx_private_queue)
[2021-04-07 08:01:54,306: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3760819c-2171-473f-8156-3191f33544b1, queue=awx_private_queue)
2021-04-07 08:01:54,311 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:01:54,310 DEBUG awx.main.dispatch task 50b660d5-8ada-438b-bc1f-d15c2b6b0190 is finished
2021-04-07 08:01:54,314 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:01:24.319201+00:00
2021-04-07 08:01:54,316 DEBUG awx.main.dispatch task d61fdd80-9c8a-4343-b446-b6edac4fd86f is finished
2021-04-07 08:01:54,320 DEBUG awx.main.dispatch delivered 3760819c-2171-473f-8156-3191f33544b1 to worker[180] qsize 0
2021-04-07 08:01:54,322 DEBUG awx.main.dispatch task 3760819c-2171-473f-8156-3191f33544b1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:01:54,324 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:01:54,327 DEBUG awx.main.dispatch task eccdf6b9-ca28-406b-9206-433f42014d69 is finished
2021-04-07 08:01:54,340 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:01:54,365 DEBUG awx.main.dispatch task 3760819c-2171-473f-8156-3191f33544b1 is finished
RESULT 2
OKREADY
2021-04-07 08:02:14,310 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0d4549a3-1a1e-4ba7-8cde-ba370bbb8c4c, queue=awx_private_queue)
[2021-04-07 08:02:14,310: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0d4549a3-1a1e-4ba7-8cde-ba370bbb8c4c, queue=awx_private_queue)
2021-04-07 08:02:14,339 DEBUG awx.main.dispatch delivered 0d4549a3-1a1e-4ba7-8cde-ba370bbb8c4c to worker[183] qsize 0
2021-04-07 08:02:14,345 DEBUG awx.main.dispatch task 0d4549a3-1a1e-4ba7-8cde-ba370bbb8c4c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:02:14,352 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:02:14,447 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:02:14,739 DEBUG awx.main.dispatch task 0d4549a3-1a1e-4ba7-8cde-ba370bbb8c4c is finished
2021-04-07 08:02:24,282 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d9625e42-2edf-4f47-bdaa-bd42043fd8e9, queue=awx_private_queue)
[2021-04-07 08:02:24,282: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d9625e42-2edf-4f47-bdaa-bd42043fd8e9, queue=awx_private_queue)
2021-04-07 08:02:24,304 DEBUG awx.main.dispatch delivered d9625e42-2edf-4f47-bdaa-bd42043fd8e9 to worker[181] qsize 0
2021-04-07 08:02:24,309 DEBUG awx.main.dispatch task d9625e42-2edf-4f47-bdaa-bd42043fd8e9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:02:24,340 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:02:24,362 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:01:54.311182+00:00
2021-04-07 08:02:24,413 DEBUG awx.main.dispatch task d9625e42-2edf-4f47-bdaa-bd42043fd8e9 is finished
2021-04-07 08:02:34,309 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bf8a21b3-cc97-4a15-b08b-40417620cbeb, queue=awx_private_queue)
[2021-04-07 08:02:34,309: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bf8a21b3-cc97-4a15-b08b-40417620cbeb, queue=awx_private_queue)
2021-04-07 08:02:34,333 DEBUG awx.main.dispatch delivered bf8a21b3-cc97-4a15-b08b-40417620cbeb to worker[182] qsize 0
2021-04-07 08:02:34,337 DEBUG awx.main.dispatch task bf8a21b3-cc97-4a15-b08b-40417620cbeb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:02:34,341 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:02:34,386 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:02:34,464 DEBUG awx.main.dispatch task bf8a21b3-cc97-4a15-b08b-40417620cbeb is finished
2021-04-07 08:02:53,990 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(548b6f35-c461-40b1-b279-9174e355d966, queue=dev-awx-0)
[2021-04-07 08:02:53,990: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(548b6f35-c461-40b1-b279-9174e355d966, queue=dev-awx-0)
2021-04-07 08:02:54,096 DEBUG awx.main.dispatch delivered 548b6f35-c461-40b1-b279-9174e355d966 to worker[180] qsize 0
2021-04-07 08:02:54,101 DEBUG awx.main.dispatch task 548b6f35-c461-40b1-b279-9174e355d966 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:02:54,105 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:02:54,190 DEBUG awx.main.dispatch task 548b6f35-c461-40b1-b279-9174e355d966 is finished
2021-04-07 08:02:54,264 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(4396ebda-c04f-45f8-8806-1514dcc26359, queue=dev-awx-0)
[2021-04-07 08:02:54,264: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(4396ebda-c04f-45f8-8806-1514dcc26359, queue=dev-awx-0)
2021-04-07 08:02:54,513 DEBUG awx.main.dispatch delivered 4396ebda-c04f-45f8-8806-1514dcc26359 to worker[180] qsize 0
2021-04-07 08:02:54,516 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b422cfc1-5200-4deb-8b4f-2218f64f8973, queue=awx_private_queue)
[2021-04-07 08:02:54,516: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b422cfc1-5200-4deb-8b4f-2218f64f8973, queue=awx_private_queue)
2021-04-07 08:02:54,518 DEBUG awx.main.dispatch task 4396ebda-c04f-45f8-8806-1514dcc26359 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:02:54,536 DEBUG awx.main.dispatch delivered b422cfc1-5200-4deb-8b4f-2218f64f8973 to worker[183] qsize 0
2021-04-07 08:02:54,538 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a36f6a78-0a68-4a9a-a314-bfb584e59bb0, queue=awx_private_queue)
[2021-04-07 08:02:54,538: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a36f6a78-0a68-4a9a-a314-bfb584e59bb0, queue=awx_private_queue)
2021-04-07 08:02:54,541 DEBUG awx.main.dispatch task b422cfc1-5200-4deb-8b4f-2218f64f8973 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:02:54,558 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:02:54,559 DEBUG awx.main.dispatch delivered a36f6a78-0a68-4a9a-a314-bfb584e59bb0 to worker[182] qsize 0
2021-04-07 08:02:54,564 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:02:24.340370+00:00
2021-04-07 08:02:54,562 DEBUG awx.main.dispatch task a36f6a78-0a68-4a9a-a314-bfb584e59bb0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:02:54,567 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:02:54,590 DEBUG awx.main.dispatch task 4396ebda-c04f-45f8-8806-1514dcc26359 is finished
2021-04-07 08:02:54,597 DEBUG awx.main.dispatch task b422cfc1-5200-4deb-8b4f-2218f64f8973 is finished
2021-04-07 08:02:54,618 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:02:54,718 DEBUG awx.main.dispatch task a36f6a78-0a68-4a9a-a314-bfb584e59bb0 is finished
RESULT 2
OKREADY
2021-04-07 08:03:14,550 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e16a2fa8-3641-4d1a-b154-79982cc500e4, queue=awx_private_queue)
[2021-04-07 08:03:14,550: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e16a2fa8-3641-4d1a-b154-79982cc500e4, queue=awx_private_queue)
2021-04-07 08:03:14,584 DEBUG awx.main.dispatch delivered e16a2fa8-3641-4d1a-b154-79982cc500e4 to worker[180] qsize 0
2021-04-07 08:03:14,595 DEBUG awx.main.dispatch task e16a2fa8-3641-4d1a-b154-79982cc500e4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:03:14,601 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:03:14,624 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:03:14,661 DEBUG awx.main.dispatch task e16a2fa8-3641-4d1a-b154-79982cc500e4 is finished
2021-04-07 08:03:24,517 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(05a5b5da-1391-4e80-b184-19ee639e4f92, queue=awx_private_queue)
[2021-04-07 08:03:24,517: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(05a5b5da-1391-4e80-b184-19ee639e4f92, queue=awx_private_queue)
2021-04-07 08:03:24,543 DEBUG awx.main.dispatch delivered 05a5b5da-1391-4e80-b184-19ee639e4f92 to worker[182] qsize 0
2021-04-07 08:03:24,550 DEBUG awx.main.dispatch task 05a5b5da-1391-4e80-b184-19ee639e4f92 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:03:24,567 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:03:24,574 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:02:54.558747+00:00
2021-04-07 08:03:24,593 DEBUG awx.main.dispatch task 05a5b5da-1391-4e80-b184-19ee639e4f92 is finished
2021-04-07 08:03:34,545 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c7dd69e4-0167-4e30-8c10-5bcb66769a3d, queue=awx_private_queue)
[2021-04-07 08:03:34,545: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c7dd69e4-0167-4e30-8c10-5bcb66769a3d, queue=awx_private_queue)
2021-04-07 08:03:34,564 DEBUG awx.main.dispatch delivered c7dd69e4-0167-4e30-8c10-5bcb66769a3d to worker[180] qsize 0
2021-04-07 08:03:34,568 DEBUG awx.main.dispatch task c7dd69e4-0167-4e30-8c10-5bcb66769a3d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:03:34,570 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:03:34,631 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:03:34,701 DEBUG awx.main.dispatch task c7dd69e4-0167-4e30-8c10-5bcb66769a3d is finished
2021-04-07 08:03:54,001 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b60edda8-d0e3-496d-98ac-82129b8026e7, queue=dev-awx-0)
[2021-04-07 08:03:54,001: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b60edda8-d0e3-496d-98ac-82129b8026e7, queue=dev-awx-0)
2021-04-07 08:03:54,098 DEBUG awx.main.dispatch delivered b60edda8-d0e3-496d-98ac-82129b8026e7 to worker[182] qsize 0
2021-04-07 08:03:54,106 DEBUG awx.main.dispatch task b60edda8-d0e3-496d-98ac-82129b8026e7 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:03:54,115 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:03:54,183 DEBUG awx.main.dispatch task b60edda8-d0e3-496d-98ac-82129b8026e7 is finished
2021-04-07 08:03:54,263 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ed5a5047-9e69-4bc1-8a08-978f6658ea65, queue=dev-awx-0)
[2021-04-07 08:03:54,263: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ed5a5047-9e69-4bc1-8a08-978f6658ea65, queue=dev-awx-0)
2021-04-07 08:03:54,276 DEBUG awx.main.dispatch delivered ed5a5047-9e69-4bc1-8a08-978f6658ea65 to worker[180] qsize 0
2021-04-07 08:03:54,288 DEBUG awx.main.dispatch task ed5a5047-9e69-4bc1-8a08-978f6658ea65 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:03:54,306 DEBUG awx.main.dispatch task ed5a5047-9e69-4bc1-8a08-978f6658ea65 is finished
2021-04-07 08:03:54,515 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7365c11e-e564-42e1-a36c-9015c9d76796, queue=awx_private_queue)
[2021-04-07 08:03:54,515: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7365c11e-e564-42e1-a36c-9015c9d76796, queue=awx_private_queue)
2021-04-07 08:03:54,712 DEBUG awx.main.dispatch delivered 7365c11e-e564-42e1-a36c-9015c9d76796 to worker[181] qsize 0
2021-04-07 08:03:54,713 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fd4496d3-1703-496b-8707-6b4266d7ef85, queue=awx_private_queue)
[2021-04-07 08:03:54,713: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fd4496d3-1703-496b-8707-6b4266d7ef85, queue=awx_private_queue)
2021-04-07 08:03:54,716 DEBUG awx.main.dispatch task 7365c11e-e564-42e1-a36c-9015c9d76796 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:03:54,725 DEBUG awx.main.dispatch delivered fd4496d3-1703-496b-8707-6b4266d7ef85 to worker[180] qsize 0
2021-04-07 08:03:54,726 DEBUG awx.main.dispatch task fd4496d3-1703-496b-8707-6b4266d7ef85 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:03:54,729 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:03:54,742 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:03:54,748 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:03:24.567466+00:00
2021-04-07 08:03:54,762 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:03:54,765 DEBUG awx.main.dispatch task 7365c11e-e564-42e1-a36c-9015c9d76796 is finished
2021-04-07 08:03:54,817 DEBUG awx.main.dispatch task fd4496d3-1703-496b-8707-6b4266d7ef85 is finished
RESULT 2
OKREADY
2021-04-07 08:04:14,725 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a968910d-9685-4b85-96f7-710f6c3ed009, queue=awx_private_queue)
[2021-04-07 08:04:14,725: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a968910d-9685-4b85-96f7-710f6c3ed009, queue=awx_private_queue)
2021-04-07 08:04:14,757 DEBUG awx.main.dispatch delivered a968910d-9685-4b85-96f7-710f6c3ed009 to worker[183] qsize 0
2021-04-07 08:04:14,762 DEBUG awx.main.dispatch task a968910d-9685-4b85-96f7-710f6c3ed009 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:04:14,768 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:04:14,798 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:04:14,846 DEBUG awx.main.dispatch task a968910d-9685-4b85-96f7-710f6c3ed009 is finished
2021-04-07 08:04:24,516 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ff35e0a0-4967-498d-996e-92a30331d0f2, queue=awx_private_queue)
[2021-04-07 08:04:24,516: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ff35e0a0-4967-498d-996e-92a30331d0f2, queue=awx_private_queue)
2021-04-07 08:04:24,548 DEBUG awx.main.dispatch delivered ff35e0a0-4967-498d-996e-92a30331d0f2 to worker[183] qsize 0
2021-04-07 08:04:24,551 DEBUG awx.main.dispatch task ff35e0a0-4967-498d-996e-92a30331d0f2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:04:24,587 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:04:24,598 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:03:54.742753+00:00
2021-04-07 08:04:24,941 DEBUG awx.main.dispatch task ff35e0a0-4967-498d-996e-92a30331d0f2 is finished
2021-04-07 08:04:34,725 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(71196f20-b786-4cfa-a55a-738853f26ebe, queue=awx_private_queue)
[2021-04-07 08:04:34,725: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(71196f20-b786-4cfa-a55a-738853f26ebe, queue=awx_private_queue)
2021-04-07 08:04:34,760 DEBUG awx.main.dispatch delivered 71196f20-b786-4cfa-a55a-738853f26ebe to worker[180] qsize 0
2021-04-07 08:04:34,766 DEBUG awx.main.dispatch task 71196f20-b786-4cfa-a55a-738853f26ebe starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:04:34,773 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:04:34,800 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:04:34,837 DEBUG awx.main.dispatch task 71196f20-b786-4cfa-a55a-738853f26ebe is finished
2021-04-07 08:04:54,012 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f3a11c45-f75d-407f-9f67-7e181585b363, queue=dev-awx-0)
[2021-04-07 08:04:54,012: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f3a11c45-f75d-407f-9f67-7e181585b363, queue=dev-awx-0)
2021-04-07 08:04:54,112 DEBUG awx.main.dispatch delivered f3a11c45-f75d-407f-9f67-7e181585b363 to worker[182] qsize 0
2021-04-07 08:04:54,264 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8e5e6687-72e0-4ff0-84ee-1c9dda122bea, queue=dev-awx-0)
[2021-04-07 08:04:54,264: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8e5e6687-72e0-4ff0-84ee-1c9dda122bea, queue=dev-awx-0)
2021-04-07 08:04:54,279 DEBUG awx.main.dispatch delivered 8e5e6687-72e0-4ff0-84ee-1c9dda122bea to worker[180] qsize 0
2021-04-07 08:04:54,282 DEBUG awx.main.dispatch task 8e5e6687-72e0-4ff0-84ee-1c9dda122bea starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:04:54,302 DEBUG awx.main.dispatch task 8e5e6687-72e0-4ff0-84ee-1c9dda122bea is finished
2021-04-07 08:04:54,116 DEBUG awx.main.dispatch task f3a11c45-f75d-407f-9f67-7e181585b363 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:04:54,319 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:04:54,336 DEBUG awx.main.dispatch task f3a11c45-f75d-407f-9f67-7e181585b363 is finished
2021-04-07 08:04:54,516 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bcf04e52-a425-4c9e-98ec-812f2d9b5346, queue=awx_private_queue)
[2021-04-07 08:04:54,516: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bcf04e52-a425-4c9e-98ec-812f2d9b5346, queue=awx_private_queue)
2021-04-07 08:04:54,541 DEBUG awx.main.dispatch delivered bcf04e52-a425-4c9e-98ec-812f2d9b5346 to worker[182] qsize 0
2021-04-07 08:04:54,544 DEBUG awx.main.dispatch task bcf04e52-a425-4c9e-98ec-812f2d9b5346 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:04:54,555 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:04:54,559 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:04:24.587771+00:00
2021-04-07 08:04:54,576 DEBUG awx.main.dispatch task bcf04e52-a425-4c9e-98ec-812f2d9b5346 is finished
2021-04-07 08:04:54,720 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6ade68d4-ac2e-4e90-8084-ce4324f8cb79, queue=awx_private_queue)
[2021-04-07 08:04:54,720: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6ade68d4-ac2e-4e90-8084-ce4324f8cb79, queue=awx_private_queue)
2021-04-07 08:04:54,739 DEBUG awx.main.dispatch delivered 6ade68d4-ac2e-4e90-8084-ce4324f8cb79 to worker[180] qsize 0
2021-04-07 08:04:54,743 DEBUG awx.main.dispatch task 6ade68d4-ac2e-4e90-8084-ce4324f8cb79 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:04:54,745 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:04:54,763 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:04:54,789 DEBUG awx.main.dispatch task 6ade68d4-ac2e-4e90-8084-ce4324f8cb79 is finished
RESULT 2
OKREADY
2021-04-07 08:05:14,760 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1ff19e5c-7d63-49b3-b27e-622981e0b858, queue=awx_private_queue)
[2021-04-07 08:05:14,760: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1ff19e5c-7d63-49b3-b27e-622981e0b858, queue=awx_private_queue)
2021-04-07 08:05:14,808 DEBUG awx.main.dispatch delivered 1ff19e5c-7d63-49b3-b27e-622981e0b858 to worker[182] qsize 0
2021-04-07 08:05:14,814 DEBUG awx.main.dispatch task 1ff19e5c-7d63-49b3-b27e-622981e0b858 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:05:14,817 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:05:14,858 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:05:14,899 DEBUG awx.main.dispatch task 1ff19e5c-7d63-49b3-b27e-622981e0b858 is finished
2021-04-07 08:05:24,519 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0715311e-d3da-4a57-820e-7795b34a516d, queue=awx_private_queue)
[2021-04-07 08:05:24,519: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0715311e-d3da-4a57-820e-7795b34a516d, queue=awx_private_queue)
2021-04-07 08:05:24,544 DEBUG awx.main.dispatch delivered 0715311e-d3da-4a57-820e-7795b34a516d to worker[180] qsize 0
2021-04-07 08:05:24,551 DEBUG awx.main.dispatch task 0715311e-d3da-4a57-820e-7795b34a516d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:05:24,754 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:05:24,783 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:04:54.555941+00:00
2021-04-07 08:05:24,846 DEBUG awx.main.dispatch task 0715311e-d3da-4a57-820e-7795b34a516d is finished
2021-04-07 08:05:34,748 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(afc835eb-d293-4f08-9ce7-201138f9c990, queue=awx_private_queue)
[2021-04-07 08:05:34,748: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(afc835eb-d293-4f08-9ce7-201138f9c990, queue=awx_private_queue)
2021-04-07 08:05:34,776 DEBUG awx.main.dispatch delivered afc835eb-d293-4f08-9ce7-201138f9c990 to worker[181] qsize 0
2021-04-07 08:05:34,783 DEBUG awx.main.dispatch task afc835eb-d293-4f08-9ce7-201138f9c990 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:05:34,791 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:05:34,821 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:05:34,858 DEBUG awx.main.dispatch task afc835eb-d293-4f08-9ce7-201138f9c990 is finished
2021-04-07 08:05:53,946 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(c7ba6a8d-a126-4c88-ad2e-c687498aeceb, queue=dev-awx-0)
[2021-04-07 08:05:53,946: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(c7ba6a8d-a126-4c88-ad2e-c687498aeceb, queue=dev-awx-0)
2021-04-07 08:05:53,968 DEBUG awx.main.dispatch delivered c7ba6a8d-a126-4c88-ad2e-c687498aeceb to worker[182] qsize 0
2021-04-07 08:05:53,973 DEBUG awx.main.dispatch task c7ba6a8d-a126-4c88-ad2e-c687498aeceb starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-04-07 08:05:53,978 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-04-07 08:05:54,009 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(20294d54-9e3e-414a-9a2f-b9d6b6cf3293, queue=dev-awx-0)
[2021-04-07 08:05:54,009: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(20294d54-9e3e-414a-9a2f-b9d6b6cf3293, queue=dev-awx-0)
2021-04-07 08:05:54,156 DEBUG awx.main.dispatch delivered 20294d54-9e3e-414a-9a2f-b9d6b6cf3293 to worker[181] qsize 0
2021-04-07 08:05:54,173 DEBUG awx.main.dispatch task 20294d54-9e3e-414a-9a2f-b9d6b6cf3293 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:05:54,175 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:05:54,263 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(59860fbe-bc09-4565-b82d-fec621bd6e62, queue=dev-awx-0)
[2021-04-07 08:05:54,263: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(59860fbe-bc09-4565-b82d-fec621bd6e62, queue=dev-awx-0)
2021-04-07 08:05:54,280 DEBUG awx.main.dispatch delivered 59860fbe-bc09-4565-b82d-fec621bd6e62 to worker[180] qsize 0
2021-04-07 08:05:54,285 DEBUG awx.main.dispatch task 59860fbe-bc09-4565-b82d-fec621bd6e62 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:05:54,708 DEBUG awx.main.dispatch task 20294d54-9e3e-414a-9a2f-b9d6b6cf3293 is finished
2021-04-07 08:05:54,822 DEBUG awx.main.dispatch task 59860fbe-bc09-4565-b82d-fec621bd6e62 is finished
2021-04-07 08:05:55,072 DEBUG awx.main.dispatch task c7ba6a8d-a126-4c88-ad2e-c687498aeceb is finished
2021-04-07 08:05:54,516 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(14f58529-a50f-4ee9-9d00-e80add4d504d, queue=awx_private_queue)
[2021-04-07 08:05:54,516: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(14f58529-a50f-4ee9-9d00-e80add4d504d, queue=awx_private_queue)
2021-04-07 08:05:55,219 DEBUG awx.main.dispatch delivered 14f58529-a50f-4ee9-9d00-e80add4d504d to worker[181] qsize 0
2021-04-07 08:05:55,220 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4a79f6d8-ad1f-4c1d-91c6-0cd08749508c, queue=awx_private_queue)
[2021-04-07 08:05:55,220: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4a79f6d8-ad1f-4c1d-91c6-0cd08749508c, queue=awx_private_queue)
2021-04-07 08:05:55,222 DEBUG awx.main.dispatch task 14f58529-a50f-4ee9-9d00-e80add4d504d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:05:55,232 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:05:55,237 DEBUG awx.main.dispatch delivered 4a79f6d8-ad1f-4c1d-91c6-0cd08749508c to worker[180] qsize 0
2021-04-07 08:05:55,239 DEBUG awx.main.dispatch task 4a79f6d8-ad1f-4c1d-91c6-0cd08749508c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:05:55,242 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:05:55,247 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:05:24.755124+00:00
2021-04-07 08:05:55,278 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:05:55,316 DEBUG awx.main.dispatch task 14f58529-a50f-4ee9-9d00-e80add4d504d is finished
2021-04-07 08:05:55,338 DEBUG awx.main.dispatch task 4a79f6d8-ad1f-4c1d-91c6-0cd08749508c is finished
RESULT 2
OKREADY
2021-04-07 08:06:15,239 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d614bf8d-977c-40e8-a2c6-607d2572ea5a, queue=awx_private_queue)
[2021-04-07 08:06:15,239: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d614bf8d-977c-40e8-a2c6-607d2572ea5a, queue=awx_private_queue)
2021-04-07 08:06:15,286 DEBUG awx.main.dispatch delivered d614bf8d-977c-40e8-a2c6-607d2572ea5a to worker[183] qsize 0
2021-04-07 08:06:15,296 DEBUG awx.main.dispatch task d614bf8d-977c-40e8-a2c6-607d2572ea5a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:06:15,304 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:06:15,333 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:06:15,373 DEBUG awx.main.dispatch task d614bf8d-977c-40e8-a2c6-607d2572ea5a is finished
2021-04-07 08:06:24,517 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(23d3957d-8098-4a0a-b5d5-3571029c690c, queue=awx_private_queue)
[2021-04-07 08:06:24,517: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(23d3957d-8098-4a0a-b5d5-3571029c690c, queue=awx_private_queue)
2021-04-07 08:06:24,535 DEBUG awx.main.dispatch delivered 23d3957d-8098-4a0a-b5d5-3571029c690c to worker[181] qsize 0
2021-04-07 08:06:24,541 DEBUG awx.main.dispatch task 23d3957d-8098-4a0a-b5d5-3571029c690c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:06:24,578 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:06:24,587 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:05:55.232852+00:00
2021-04-07 08:06:24,618 DEBUG awx.main.dispatch task 23d3957d-8098-4a0a-b5d5-3571029c690c is finished
2021-04-07 08:06:35,232 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(492bcdad-f919-4881-a9ae-82685dd97c7a, queue=awx_private_queue)
[2021-04-07 08:06:35,232: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(492bcdad-f919-4881-a9ae-82685dd97c7a, queue=awx_private_queue)
2021-04-07 08:06:35,250 DEBUG awx.main.dispatch delivered 492bcdad-f919-4881-a9ae-82685dd97c7a to worker[181] qsize 0
2021-04-07 08:06:35,252 DEBUG awx.main.dispatch task 492bcdad-f919-4881-a9ae-82685dd97c7a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:06:35,254 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:06:35,275 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:06:35,303 DEBUG awx.main.dispatch task 492bcdad-f919-4881-a9ae-82685dd97c7a is finished
2021-04-07 08:06:54,010 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(284f1932-e771-4690-8b6e-982fb2005c98, queue=dev-awx-0)
[2021-04-07 08:06:54,010: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(284f1932-e771-4690-8b6e-982fb2005c98, queue=dev-awx-0)
2021-04-07 08:06:54,047 DEBUG awx.main.dispatch delivered 284f1932-e771-4690-8b6e-982fb2005c98 to worker[182] qsize 0
2021-04-07 08:06:54,053 DEBUG awx.main.dispatch task 284f1932-e771-4690-8b6e-982fb2005c98 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:06:54,060 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:06:54,101 DEBUG awx.main.dispatch task 284f1932-e771-4690-8b6e-982fb2005c98 is finished
2021-04-07 08:06:54,263 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ae754a3c-7277-4808-a875-c8824081d835, queue=dev-awx-0)
[2021-04-07 08:06:54,263: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ae754a3c-7277-4808-a875-c8824081d835, queue=dev-awx-0)
2021-04-07 08:06:54,277 DEBUG awx.main.dispatch delivered ae754a3c-7277-4808-a875-c8824081d835 to worker[180] qsize 0
2021-04-07 08:06:54,280 DEBUG awx.main.dispatch task ae754a3c-7277-4808-a875-c8824081d835 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:06:54,306 DEBUG awx.main.dispatch task ae754a3c-7277-4808-a875-c8824081d835 is finished
2021-04-07 08:06:54,516 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f09e75ba-6958-44db-8ba9-42b45a94c11b, queue=awx_private_queue)
[2021-04-07 08:06:54,516: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f09e75ba-6958-44db-8ba9-42b45a94c11b, queue=awx_private_queue)
2021-04-07 08:06:54,529 DEBUG awx.main.dispatch delivered f09e75ba-6958-44db-8ba9-42b45a94c11b to worker[183] qsize 0
2021-04-07 08:06:54,531 DEBUG awx.main.dispatch task f09e75ba-6958-44db-8ba9-42b45a94c11b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:06:54,551 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:06:54,554 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:06:24.579126+00:00
2021-04-07 08:06:54,573 DEBUG awx.main.dispatch task f09e75ba-6958-44db-8ba9-42b45a94c11b is finished
2021-04-07 08:06:55,231 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0f19965e-73f9-4278-931a-37d2d2f81c7c, queue=awx_private_queue)
[2021-04-07 08:06:55,231: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0f19965e-73f9-4278-931a-37d2d2f81c7c, queue=awx_private_queue)
2021-04-07 08:06:55,433 DEBUG awx.main.dispatch delivered 0f19965e-73f9-4278-931a-37d2d2f81c7c to worker[183] qsize 0
2021-04-07 08:06:55,435 DEBUG awx.main.dispatch task 0f19965e-73f9-4278-931a-37d2d2f81c7c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:06:55,436 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:06:55,450 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:06:55,475 DEBUG awx.main.dispatch task 0f19965e-73f9-4278-931a-37d2d2f81c7c is finished
RESULT 2
OKREADY
2021-04-07 08:07:15,238 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4cee4a81-d8a3-4afc-9645-e3390cf5f58f, queue=awx_private_queue)
[2021-04-07 08:07:15,238: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4cee4a81-d8a3-4afc-9645-e3390cf5f58f, queue=awx_private_queue)
2021-04-07 08:07:15,274 DEBUG awx.main.dispatch delivered 4cee4a81-d8a3-4afc-9645-e3390cf5f58f to worker[181] qsize 0
2021-04-07 08:07:15,287 DEBUG awx.main.dispatch task 4cee4a81-d8a3-4afc-9645-e3390cf5f58f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:07:15,297 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:07:15,328 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:07:15,361 DEBUG awx.main.dispatch task 4cee4a81-d8a3-4afc-9645-e3390cf5f58f is finished
2021-04-07 08:07:24,521 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(92bea155-9829-4b32-b630-ed7c45f3968e, queue=awx_private_queue)
[2021-04-07 08:07:24,521: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(92bea155-9829-4b32-b630-ed7c45f3968e, queue=awx_private_queue)
2021-04-07 08:07:24,545 DEBUG awx.main.dispatch delivered 92bea155-9829-4b32-b630-ed7c45f3968e to worker[183] qsize 0
2021-04-07 08:07:24,550 DEBUG awx.main.dispatch task 92bea155-9829-4b32-b630-ed7c45f3968e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:07:24,565 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:07:24,571 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:06:54.551582+00:00
2021-04-07 08:07:24,586 DEBUG awx.main.dispatch task 92bea155-9829-4b32-b630-ed7c45f3968e is finished
2021-04-07 08:07:35,232 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ed656678-0929-4100-8551-97c3ffbda4a8, queue=awx_private_queue)
[2021-04-07 08:07:35,232: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ed656678-0929-4100-8551-97c3ffbda4a8, queue=awx_private_queue)
2021-04-07 08:07:35,257 DEBUG awx.main.dispatch delivered ed656678-0929-4100-8551-97c3ffbda4a8 to worker[180] qsize 0
2021-04-07 08:07:35,262 DEBUG awx.main.dispatch task ed656678-0929-4100-8551-97c3ffbda4a8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:07:35,268 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:07:35,298 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:07:35,329 DEBUG awx.main.dispatch task ed656678-0929-4100-8551-97c3ffbda4a8 is finished
2021-04-07 08:07:54,009 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(dfe3b88b-ae87-4254-bfd4-e7c8f2a12e18, queue=dev-awx-0)
[2021-04-07 08:07:54,009: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(dfe3b88b-ae87-4254-bfd4-e7c8f2a12e18, queue=dev-awx-0)
2021-04-07 08:07:54,058 DEBUG awx.main.dispatch delivered dfe3b88b-ae87-4254-bfd4-e7c8f2a12e18 to worker[182] qsize 0
2021-04-07 08:07:54,063 DEBUG awx.main.dispatch task dfe3b88b-ae87-4254-bfd4-e7c8f2a12e18 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:07:54,071 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:07:54,182 DEBUG awx.main.dispatch task dfe3b88b-ae87-4254-bfd4-e7c8f2a12e18 is finished
2021-04-07 08:07:54,264 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3742f08f-5746-445f-bc88-10213cd1dc0d, queue=dev-awx-0)
[2021-04-07 08:07:54,264: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3742f08f-5746-445f-bc88-10213cd1dc0d, queue=dev-awx-0)
2021-04-07 08:07:54,291 DEBUG awx.main.dispatch delivered 3742f08f-5746-445f-bc88-10213cd1dc0d to worker[180] qsize 0
2021-04-07 08:07:54,295 DEBUG awx.main.dispatch task 3742f08f-5746-445f-bc88-10213cd1dc0d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:07:54,329 DEBUG awx.main.dispatch task 3742f08f-5746-445f-bc88-10213cd1dc0d is finished
2021-04-07 08:07:54,516 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3cca38c7-75df-4055-bf72-bb42cdc1c84f, queue=awx_private_queue)
[2021-04-07 08:07:54,516: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3cca38c7-75df-4055-bf72-bb42cdc1c84f, queue=awx_private_queue)
2021-04-07 08:07:54,532 DEBUG awx.main.dispatch delivered 3cca38c7-75df-4055-bf72-bb42cdc1c84f to worker[180] qsize 0
2021-04-07 08:07:54,534 DEBUG awx.main.dispatch task 3cca38c7-75df-4055-bf72-bb42cdc1c84f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:07:54,549 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:07:54,552 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:07:24.565186+00:00
2021-04-07 08:07:54,568 DEBUG awx.main.dispatch task 3cca38c7-75df-4055-bf72-bb42cdc1c84f is finished
2021-04-07 08:07:55,231 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fa5f7b17-a5f9-4b0d-a8f6-8df6a6bfb31b, queue=awx_private_queue)
[2021-04-07 08:07:55,231: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fa5f7b17-a5f9-4b0d-a8f6-8df6a6bfb31b, queue=awx_private_queue)
2021-04-07 08:07:55,453 DEBUG awx.main.dispatch delivered fa5f7b17-a5f9-4b0d-a8f6-8df6a6bfb31b to worker[183] qsize 0
2021-04-07 08:07:55,457 DEBUG awx.main.dispatch task fa5f7b17-a5f9-4b0d-a8f6-8df6a6bfb31b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:07:55,458 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:07:55,486 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:07:55,517 DEBUG awx.main.dispatch task fa5f7b17-a5f9-4b0d-a8f6-8df6a6bfb31b is finished
RESULT 2
OKREADY
2021-04-07 08:08:15,247 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bc1c7bc1-95ee-40c7-86b8-3e5461e4a5dc, queue=awx_private_queue)
[2021-04-07 08:08:15,247: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bc1c7bc1-95ee-40c7-86b8-3e5461e4a5dc, queue=awx_private_queue)
2021-04-07 08:08:15,291 DEBUG awx.main.dispatch delivered bc1c7bc1-95ee-40c7-86b8-3e5461e4a5dc to worker[181] qsize 0
2021-04-07 08:08:15,300 DEBUG awx.main.dispatch task bc1c7bc1-95ee-40c7-86b8-3e5461e4a5dc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:08:15,308 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:08:15,337 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:08:15,372 DEBUG awx.main.dispatch task bc1c7bc1-95ee-40c7-86b8-3e5461e4a5dc is finished
2021-04-07 08:08:24,518 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1affff1e-152f-4ae8-a58d-02350cfb8778, queue=awx_private_queue)
[2021-04-07 08:08:24,518: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1affff1e-152f-4ae8-a58d-02350cfb8778, queue=awx_private_queue)
2021-04-07 08:08:24,540 DEBUG awx.main.dispatch delivered 1affff1e-152f-4ae8-a58d-02350cfb8778 to worker[183] qsize 0
2021-04-07 08:08:24,546 DEBUG awx.main.dispatch task 1affff1e-152f-4ae8-a58d-02350cfb8778 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:08:24,571 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:08:24,576 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:07:54.549818+00:00
2021-04-07 08:08:24,593 DEBUG awx.main.dispatch task 1affff1e-152f-4ae8-a58d-02350cfb8778 is finished
2021-04-07 08:08:35,246 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(83070a2f-9313-4ea6-b5e8-d7b9c60590a3, queue=awx_private_queue)
[2021-04-07 08:08:35,246: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(83070a2f-9313-4ea6-b5e8-d7b9c60590a3, queue=awx_private_queue)
2021-04-07 08:08:35,263 DEBUG awx.main.dispatch delivered 83070a2f-9313-4ea6-b5e8-d7b9c60590a3 to worker[181] qsize 0
2021-04-07 08:08:35,270 DEBUG awx.main.dispatch task 83070a2f-9313-4ea6-b5e8-d7b9c60590a3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:08:35,273 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:08:35,428 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:08:35,482 DEBUG awx.main.dispatch task 83070a2f-9313-4ea6-b5e8-d7b9c60590a3 is finished
2021-04-07 08:08:54,017 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b449fd2b-10b4-4561-898a-b2b2fb65985f, queue=dev-awx-0)
[2021-04-07 08:08:54,017: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b449fd2b-10b4-4561-898a-b2b2fb65985f, queue=dev-awx-0)
2021-04-07 08:08:54,265 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(bd35c4ab-a976-40e9-9d2f-658a04d416b0, queue=dev-awx-0)
[2021-04-07 08:08:54,265: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(bd35c4ab-a976-40e9-9d2f-658a04d416b0, queue=dev-awx-0)
2021-04-07 08:08:54,517 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(50163464-82b1-4388-875d-3b50b368ae66, queue=awx_private_queue)
[2021-04-07 08:08:54,517: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(50163464-82b1-4388-875d-3b50b368ae66, queue=awx_private_queue)
2021-04-07 08:08:54,845 DEBUG awx.main.dispatch delivered b449fd2b-10b4-4561-898a-b2b2fb65985f to worker[182] qsize 0
2021-04-07 08:08:54,849 DEBUG awx.main.dispatch delivered bd35c4ab-a976-40e9-9d2f-658a04d416b0 to worker[181] qsize 0
2021-04-07 08:08:54,852 DEBUG awx.main.dispatch delivered 50163464-82b1-4388-875d-3b50b368ae66 to worker[183] qsize 0
2021-04-07 08:08:54,853 DEBUG awx.main.dispatch task bd35c4ab-a976-40e9-9d2f-658a04d416b0 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:08:54,857 DEBUG awx.main.dispatch task 50163464-82b1-4388-875d-3b50b368ae66 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:08:54,861 DEBUG awx.main.dispatch task b449fd2b-10b4-4561-898a-b2b2fb65985f starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:08:54,871 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:08:55,245 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7627fe69-4e75-4ee6-8a28-6e46e22a0ba4, queue=awx_private_queue)
[2021-04-07 08:08:55,245: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7627fe69-4e75-4ee6-8a28-6e46e22a0ba4, queue=awx_private_queue)
2021-04-07 08:08:55,265 DEBUG awx.main.dispatch delivered 7627fe69-4e75-4ee6-8a28-6e46e22a0ba4 to worker[180] qsize 0
2021-04-07 08:08:55,328 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:08:55,339 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:08:24.571659+00:00
2021-04-07 08:08:55,273 DEBUG awx.main.dispatch task 7627fe69-4e75-4ee6-8a28-6e46e22a0ba4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:08:55,358 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:08:55,342 DEBUG awx.main.dispatch task bd35c4ab-a976-40e9-9d2f-658a04d416b0 is finished
2021-04-07 08:08:55,368 DEBUG awx.main.dispatch task 50163464-82b1-4388-875d-3b50b368ae66 is finished
2021-04-07 08:08:55,385 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:08:55,559 DEBUG awx.main.dispatch task b449fd2b-10b4-4561-898a-b2b2fb65985f is finished
2021-04-07 08:08:55,588 DEBUG awx.main.dispatch task 7627fe69-4e75-4ee6-8a28-6e46e22a0ba4 is finished
RESULT 2
OKREADY
2021-04-07 08:09:15,260 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(523908a6-9bbb-4c81-a93c-9ccd743dc645, queue=awx_private_queue)
[2021-04-07 08:09:15,260: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(523908a6-9bbb-4c81-a93c-9ccd743dc645, queue=awx_private_queue)
2021-04-07 08:09:15,288 DEBUG awx.main.dispatch delivered 523908a6-9bbb-4c81-a93c-9ccd743dc645 to worker[180] qsize 0
2021-04-07 08:09:15,527 DEBUG awx.main.dispatch task 523908a6-9bbb-4c81-a93c-9ccd743dc645 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:09:15,532 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:09:15,558 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:09:15,587 DEBUG awx.main.dispatch task 523908a6-9bbb-4c81-a93c-9ccd743dc645 is finished
2021-04-07 08:09:24,519 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d06f7142-e0a3-48ca-93c3-6c064887b37a, queue=awx_private_queue)
[2021-04-07 08:09:24,519: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d06f7142-e0a3-48ca-93c3-6c064887b37a, queue=awx_private_queue)
2021-04-07 08:09:24,549 DEBUG awx.main.dispatch delivered d06f7142-e0a3-48ca-93c3-6c064887b37a to worker[182] qsize 0
2021-04-07 08:09:24,555 DEBUG awx.main.dispatch task d06f7142-e0a3-48ca-93c3-6c064887b37a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:09:24,585 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:09:24,595 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:08:55.328706+00:00
2021-04-07 08:09:24,638 DEBUG awx.main.dispatch task d06f7142-e0a3-48ca-93c3-6c064887b37a is finished
2021-04-07 08:09:35,257 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(93ebbc7e-8d1a-4ef3-b848-5e8654ea5981, queue=awx_private_queue)
[2021-04-07 08:09:35,257: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(93ebbc7e-8d1a-4ef3-b848-5e8654ea5981, queue=awx_private_queue)
2021-04-07 08:09:35,277 DEBUG awx.main.dispatch delivered 93ebbc7e-8d1a-4ef3-b848-5e8654ea5981 to worker[181] qsize 0
2021-04-07 08:09:35,282 DEBUG awx.main.dispatch task 93ebbc7e-8d1a-4ef3-b848-5e8654ea5981 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:09:35,287 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:09:35,328 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:09:35,723 DEBUG awx.main.dispatch task 93ebbc7e-8d1a-4ef3-b848-5e8654ea5981 is finished
2021-04-07 08:09:54,025 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(7910f51f-bc33-4f2f-b59f-b6fab7cef4b0, queue=dev-awx-0)
[2021-04-07 08:09:54,025: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(7910f51f-bc33-4f2f-b59f-b6fab7cef4b0, queue=dev-awx-0)
2021-04-07 08:09:54,264 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8c44023f-9e4e-49ef-8122-2472ae68bd13, queue=dev-awx-0)
[2021-04-07 08:09:54,264: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8c44023f-9e4e-49ef-8122-2472ae68bd13, queue=dev-awx-0)
2021-04-07 08:09:54,336 DEBUG awx.main.dispatch delivered 7910f51f-bc33-4f2f-b59f-b6fab7cef4b0 to worker[180] qsize 0
2021-04-07 08:09:54,338 DEBUG awx.main.dispatch delivered 8c44023f-9e4e-49ef-8122-2472ae68bd13 to worker[183] qsize 0
2021-04-07 08:09:54,340 DEBUG awx.main.dispatch task 7910f51f-bc33-4f2f-b59f-b6fab7cef4b0 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:09:54,343 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:09:54,342 DEBUG awx.main.dispatch task 8c44023f-9e4e-49ef-8122-2472ae68bd13 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:09:54,378 DEBUG awx.main.dispatch task 8c44023f-9e4e-49ef-8122-2472ae68bd13 is finished
2021-04-07 08:09:54,517 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(97ba7902-09ca-4933-8524-124a670179f0, queue=awx_private_queue)
[2021-04-07 08:09:54,517: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(97ba7902-09ca-4933-8524-124a670179f0, queue=awx_private_queue)
2021-04-07 08:09:54,534 DEBUG awx.main.dispatch delivered 97ba7902-09ca-4933-8524-124a670179f0 to worker[183] qsize 0
2021-04-07 08:09:54,537 DEBUG awx.main.dispatch task 97ba7902-09ca-4933-8524-124a670179f0 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:09:54,703 DEBUG awx.main.dispatch task 7910f51f-bc33-4f2f-b59f-b6fab7cef4b0 is finished
2021-04-07 08:09:54,794 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:09:54,799 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:09:24.585891+00:00
2021-04-07 08:09:54,881 DEBUG awx.main.dispatch task 97ba7902-09ca-4933-8524-124a670179f0 is finished
2021-04-07 08:09:55,257 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c4c34707-1b0e-498f-ae5d-93e970259f48, queue=awx_private_queue)
[2021-04-07 08:09:55,257: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c4c34707-1b0e-498f-ae5d-93e970259f48, queue=awx_private_queue)
2021-04-07 08:09:55,289 DEBUG awx.main.dispatch delivered c4c34707-1b0e-498f-ae5d-93e970259f48 to worker[181] qsize 0
2021-04-07 08:09:55,292 DEBUG awx.main.dispatch task c4c34707-1b0e-498f-ae5d-93e970259f48 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:09:55,295 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:09:55,343 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:09:55,630 DEBUG awx.main.dispatch task c4c34707-1b0e-498f-ae5d-93e970259f48 is finished
RESULT 2
OKREADY
2021-04-07 08:10:15,270 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0838b26b-490f-4acc-badb-b51b782de8dd, queue=awx_private_queue)
[2021-04-07 08:10:15,270: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0838b26b-490f-4acc-badb-b51b782de8dd, queue=awx_private_queue)
2021-04-07 08:10:15,303 DEBUG awx.main.dispatch delivered 0838b26b-490f-4acc-badb-b51b782de8dd to worker[183] qsize 0
2021-04-07 08:10:15,312 DEBUG awx.main.dispatch task 0838b26b-490f-4acc-badb-b51b782de8dd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:10:15,315 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:10:17,155 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:10:18,467 DEBUG awx.main.dispatch task 0838b26b-490f-4acc-badb-b51b782de8dd is finished
2021-04-07 08:10:24,520 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4b5978ef-ceae-4676-ab47-06e3db3af8ae, queue=awx_private_queue)
[2021-04-07 08:10:24,520: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4b5978ef-ceae-4676-ab47-06e3db3af8ae, queue=awx_private_queue)
2021-04-07 08:10:24,539 DEBUG awx.main.dispatch delivered 4b5978ef-ceae-4676-ab47-06e3db3af8ae to worker[183] qsize 0
2021-04-07 08:10:24,542 DEBUG awx.main.dispatch task 4b5978ef-ceae-4676-ab47-06e3db3af8ae starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:10:28,007 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:10:28,051 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:09:54.795266+00:00
2021-04-07 08:10:30,870 DEBUG awx.main.dispatch task 4b5978ef-ceae-4676-ab47-06e3db3af8ae is finished
2021-04-07 08:10:35,264 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5e1fa1b0-6613-4d1c-8476-434f13af051f, queue=awx_private_queue)
[2021-04-07 08:10:35,264: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5e1fa1b0-6613-4d1c-8476-434f13af051f, queue=awx_private_queue)
2021-04-07 08:10:35,298 DEBUG awx.main.dispatch delivered 5e1fa1b0-6613-4d1c-8476-434f13af051f to worker[181] qsize 0
2021-04-07 08:10:35,303 DEBUG awx.main.dispatch task 5e1fa1b0-6613-4d1c-8476-434f13af051f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:10:50,404 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:10:50,428 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:10:50,718 DEBUG awx.main.dispatch task 5e1fa1b0-6613-4d1c-8476-434f13af051f is finished
2021-04-07 08:10:54,035 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(aae244b1-e430-4a2f-a703-734f5627199d, queue=dev-awx-0)
[2021-04-07 08:10:54,035: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(aae244b1-e430-4a2f-a703-734f5627199d, queue=dev-awx-0)
2021-04-07 08:10:54,264 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(69a58e9e-9079-417a-ab84-eefb7c253adc, queue=dev-awx-0)
[2021-04-07 08:10:54,264: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(69a58e9e-9079-417a-ab84-eefb7c253adc, queue=dev-awx-0)
2021-04-07 08:10:54,518 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(cfa88d02-11b9-4f91-bfd3-3bcb9069565e, queue=awx_private_queue)
[2021-04-07 08:10:54,518: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(cfa88d02-11b9-4f91-bfd3-3bcb9069565e, queue=awx_private_queue)
2021-04-07 08:10:55,148 DEBUG awx.main.dispatch delivered aae244b1-e430-4a2f-a703-734f5627199d to worker[181] qsize 0
2021-04-07 08:10:55,150 DEBUG awx.main.dispatch delivered 69a58e9e-9079-417a-ab84-eefb7c253adc to worker[182] qsize 0
2021-04-07 08:10:55,150 DEBUG awx.main.dispatch task aae244b1-e430-4a2f-a703-734f5627199d starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:10:55,152 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:10:55,153 DEBUG awx.main.dispatch delivered cfa88d02-11b9-4f91-bfd3-3bcb9069565e to worker[183] qsize 0
2021-04-07 08:10:55,156 DEBUG awx.main.dispatch task cfa88d02-11b9-4f91-bfd3-3bcb9069565e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:10:55,155 DEBUG awx.main.dispatch task 69a58e9e-9079-417a-ab84-eefb7c253adc starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:10:55,263 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f7d65176-b870-4cfc-9ed5-917093b3625f, queue=awx_private_queue)
[2021-04-07 08:10:55,263: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f7d65176-b870-4cfc-9ed5-917093b3625f, queue=awx_private_queue)
2021-04-07 08:10:55,278 DEBUG awx.main.dispatch delivered f7d65176-b870-4cfc-9ed5-917093b3625f to worker[180] qsize 0
2021-04-07 08:10:55,282 DEBUG awx.main.dispatch task f7d65176-b870-4cfc-9ed5-917093b3625f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:10:55,288 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:10:55,302 DEBUG awx.main.dispatch task 69a58e9e-9079-417a-ab84-eefb7c253adc is finished
2021-04-07 08:10:55,331 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:10:55,332 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:10:55,335 DEBUG awx.main.dispatch task aae244b1-e430-4a2f-a703-734f5627199d is finished
2021-04-07 08:10:55,587 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:10:28.007674+00:00
2021-04-07 08:10:55,637 DEBUG awx.main.dispatch task cfa88d02-11b9-4f91-bfd3-3bcb9069565e is finished
2021-04-07 08:10:55,991 DEBUG awx.main.dispatch task f7d65176-b870-4cfc-9ed5-917093b3625f is finished
RESULT 2
OKREADY
2021-04-07 08:11:15,281 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3cdbb584-056a-4e6b-b63e-afa1cda3533f, queue=awx_private_queue)
[2021-04-07 08:11:15,281: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3cdbb584-056a-4e6b-b63e-afa1cda3533f, queue=awx_private_queue)
2021-04-07 08:11:15,309 DEBUG awx.main.dispatch delivered 3cdbb584-056a-4e6b-b63e-afa1cda3533f to worker[182] qsize 0
2021-04-07 08:11:15,315 DEBUG awx.main.dispatch task 3cdbb584-056a-4e6b-b63e-afa1cda3533f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:11:15,318 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:11:23,240 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:11:23,375 DEBUG awx.main.dispatch task 3cdbb584-056a-4e6b-b63e-afa1cda3533f is finished
2021-04-07 08:11:24,521 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ff3bc604-f23d-4233-a925-295a3089d105, queue=awx_private_queue)
[2021-04-07 08:11:24,521: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ff3bc604-f23d-4233-a925-295a3089d105, queue=awx_private_queue)
2021-04-07 08:11:24,557 DEBUG awx.main.dispatch delivered ff3bc604-f23d-4233-a925-295a3089d105 to worker[182] qsize 0
2021-04-07 08:11:24,565 DEBUG awx.main.dispatch task ff3bc604-f23d-4233-a925-295a3089d105 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:11:24,917 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:11:24,925 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:10:55.331639+00:00
2021-04-07 08:11:24,973 DEBUG awx.main.dispatch task ff3bc604-f23d-4233-a925-295a3089d105 is finished
2021-04-07 08:11:35,282 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5802bce9-3c5b-4179-be60-85e2c30bd5e1, queue=awx_private_queue)
[2021-04-07 08:11:35,282: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5802bce9-3c5b-4179-be60-85e2c30bd5e1, queue=awx_private_queue)
2021-04-07 08:11:35,307 DEBUG awx.main.dispatch delivered 5802bce9-3c5b-4179-be60-85e2c30bd5e1 to worker[180] qsize 0
2021-04-07 08:11:35,315 DEBUG awx.main.dispatch task 5802bce9-3c5b-4179-be60-85e2c30bd5e1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:11:35,320 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:11:49,075 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:11:50,191 DEBUG awx.main.dispatch task 5802bce9-3c5b-4179-be60-85e2c30bd5e1 is finished
RESULT 2
OKREADY
2021-04-07 08:11:54,045 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(cb577d68-5163-4e57-bc65-a192f893dc5a, queue=dev-awx-0)
[2021-04-07 08:11:54,045: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(cb577d68-5163-4e57-bc65-a192f893dc5a, queue=dev-awx-0)
2021-04-07 08:12:05,328 DEBUG awx.main.dispatch delivered cb577d68-5163-4e57-bc65-a192f893dc5a to worker[182] qsize 0
2021-04-07 08:12:05,135 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3ff6ba0c-38cc-4349-a19f-f88564afc5d9, queue=dev-awx-0)
[2021-04-07 08:12:05,135: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3ff6ba0c-38cc-4349-a19f-f88564afc5d9, queue=dev-awx-0)
2021-04-07 08:12:05,405 DEBUG awx.main.dispatch delivered 3ff6ba0c-38cc-4349-a19f-f88564afc5d9 to worker[181] qsize 0
2021-04-07 08:12:05,406 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5b0c9a09-4016-4738-838a-c9bf91ce7115, queue=awx_private_queue)
[2021-04-07 08:12:05,406: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5b0c9a09-4016-4738-838a-c9bf91ce7115, queue=awx_private_queue)
2021-04-07 08:12:05,409 DEBUG awx.main.dispatch task 3ff6ba0c-38cc-4349-a19f-f88564afc5d9 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:12:05,429 DEBUG awx.main.dispatch delivered 5b0c9a09-4016-4738-838a-c9bf91ce7115 to worker[180] qsize 0
2021-04-07 08:12:05,431 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(64da5a9d-61c9-4c74-9adf-2bb0c76b3d23, queue=awx_private_queue)
[2021-04-07 08:12:05,431: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(64da5a9d-61c9-4c74-9adf-2bb0c76b3d23, queue=awx_private_queue)
2021-04-07 08:12:05,431 DEBUG awx.main.dispatch task 5b0c9a09-4016-4738-838a-c9bf91ce7115 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:12:05,472 DEBUG awx.main.dispatch delivered 64da5a9d-61c9-4c74-9adf-2bb0c76b3d23 to worker[183] qsize 0
2021-04-07 08:12:05,478 DEBUG awx.main.dispatch task 64da5a9d-61c9-4c74-9adf-2bb0c76b3d23 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:12:05,485 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:12:05,507 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:12:05,522 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:11:24.918260+00:00
2021-04-07 08:12:05,521 DEBUG awx.main.dispatch task 3ff6ba0c-38cc-4349-a19f-f88564afc5d9 is finished
2021-04-07 08:12:05,529 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:12:05,619 DEBUG awx.main.dispatch task 5b0c9a09-4016-4738-838a-c9bf91ce7115 is finished
2021-04-07 08:12:05,627 DEBUG awx.main.dispatch task 64da5a9d-61c9-4c74-9adf-2bb0c76b3d23 is finished
2021-04-07 08:12:05,334 DEBUG awx.main.dispatch task cb577d68-5163-4e57-bc65-a192f893dc5a starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:12:05,659 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:12:05,744 DEBUG awx.main.dispatch task cb577d68-5163-4e57-bc65-a192f893dc5a is finished
2021-04-07 08:12:25,441 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c0d46ce0-044f-448c-bef3-e3f8ab091fb8, queue=awx_private_queue)
[2021-04-07 08:12:25,441: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c0d46ce0-044f-448c-bef3-e3f8ab091fb8, queue=awx_private_queue)
2021-04-07 08:12:25,462 DEBUG awx.main.dispatch delivered c0d46ce0-044f-448c-bef3-e3f8ab091fb8 to worker[182] qsize 0
2021-04-07 08:12:25,466 DEBUG awx.main.dispatch task c0d46ce0-044f-448c-bef3-e3f8ab091fb8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:12:25,468 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:12:25,502 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:12:25,581 DEBUG awx.main.dispatch task c0d46ce0-044f-448c-bef3-e3f8ab091fb8 is finished
2021-04-07 08:12:35,412 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(accd271a-cb6e-43e7-9939-f8afbda24af0, queue=awx_private_queue)
[2021-04-07 08:12:35,412: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(accd271a-cb6e-43e7-9939-f8afbda24af0, queue=awx_private_queue)
2021-04-07 08:12:35,437 DEBUG awx.main.dispatch delivered accd271a-cb6e-43e7-9939-f8afbda24af0 to worker[181] qsize 0
2021-04-07 08:12:35,445 DEBUG awx.main.dispatch task accd271a-cb6e-43e7-9939-f8afbda24af0 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:12:35,462 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:12:35,468 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:12:05.507817+00:00
2021-04-07 08:12:35,489 DEBUG awx.main.dispatch task accd271a-cb6e-43e7-9939-f8afbda24af0 is finished
2021-04-07 08:12:45,443 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(be453995-c92a-4aa4-871b-2d9054dbb56e, queue=awx_private_queue)
[2021-04-07 08:12:45,443: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(be453995-c92a-4aa4-871b-2d9054dbb56e, queue=awx_private_queue)
2021-04-07 08:12:45,462 DEBUG awx.main.dispatch delivered be453995-c92a-4aa4-871b-2d9054dbb56e to worker[181] qsize 0
2021-04-07 08:12:45,466 DEBUG awx.main.dispatch task be453995-c92a-4aa4-871b-2d9054dbb56e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:12:45,468 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:12:45,534 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:12:45,583 DEBUG awx.main.dispatch task be453995-c92a-4aa4-871b-2d9054dbb56e is finished
2021-04-07 08:12:54,046 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(65ad2917-09db-4fdd-a554-66dcf76186d8, queue=dev-awx-0)
[2021-04-07 08:12:54,046: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(65ad2917-09db-4fdd-a554-66dcf76186d8, queue=dev-awx-0)
2021-04-07 08:12:54,325 DEBUG awx.main.dispatch delivered 65ad2917-09db-4fdd-a554-66dcf76186d8 to worker[181] qsize 0
2021-04-07 08:12:54,328 DEBUG awx.main.dispatch task 65ad2917-09db-4fdd-a554-66dcf76186d8 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:12:54,330 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:12:54,369 DEBUG awx.main.dispatch task 65ad2917-09db-4fdd-a554-66dcf76186d8 is finished
RESULT 2
OKREADY
2021-04-07 08:13:05,136 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a325794e-aa30-4c19-9f78-49a496c72f65, queue=dev-awx-0)
[2021-04-07 08:13:05,136: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a325794e-aa30-4c19-9f78-49a496c72f65, queue=dev-awx-0)
2021-04-07 08:13:05,166 DEBUG awx.main.dispatch delivered a325794e-aa30-4c19-9f78-49a496c72f65 to worker[181] qsize 0
2021-04-07 08:13:05,169 DEBUG awx.main.dispatch task a325794e-aa30-4c19-9f78-49a496c72f65 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:13:05,218 DEBUG awx.main.dispatch task a325794e-aa30-4c19-9f78-49a496c72f65 is finished
2021-04-07 08:13:05,409 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a36406d7-017a-40e4-81fe-0cc5c76717cb, queue=awx_private_queue)
[2021-04-07 08:13:05,409: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a36406d7-017a-40e4-81fe-0cc5c76717cb, queue=awx_private_queue)
2021-04-07 08:13:05,443 DEBUG awx.main.dispatch delivered a36406d7-017a-40e4-81fe-0cc5c76717cb to worker[180] qsize 0
2021-04-07 08:13:05,448 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b7c13744-4098-4bc6-b46b-d7aa5b66c0c6, queue=awx_private_queue)
[2021-04-07 08:13:05,448: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b7c13744-4098-4bc6-b46b-d7aa5b66c0c6, queue=awx_private_queue)
2021-04-07 08:13:05,449 DEBUG awx.main.dispatch task a36406d7-017a-40e4-81fe-0cc5c76717cb starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:13:05,468 DEBUG awx.main.dispatch delivered b7c13744-4098-4bc6-b46b-d7aa5b66c0c6 to worker[181] qsize 0
2021-04-07 08:13:05,471 DEBUG awx.main.dispatch task b7c13744-4098-4bc6-b46b-d7aa5b66c0c6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:13:05,475 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:13:05,475 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:13:05,485 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:12:35.462465+00:00
2021-04-07 08:13:05,497 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:13:05,506 DEBUG awx.main.dispatch task a36406d7-017a-40e4-81fe-0cc5c76717cb is finished
2021-04-07 08:13:05,534 DEBUG awx.main.dispatch task b7c13744-4098-4bc6-b46b-d7aa5b66c0c6 is finished
2021-04-07 08:13:25,458 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0497a8da-1151-4843-891d-888c5057a133, queue=awx_private_queue)
[2021-04-07 08:13:25,458: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0497a8da-1151-4843-891d-888c5057a133, queue=awx_private_queue)
2021-04-07 08:13:25,483 DEBUG awx.main.dispatch delivered 0497a8da-1151-4843-891d-888c5057a133 to worker[181] qsize 0
2021-04-07 08:13:26,077 DEBUG awx.main.dispatch task 0497a8da-1151-4843-891d-888c5057a133 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:13:26,079 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:13:28,524 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:13:29,206 DEBUG awx.main.dispatch task 0497a8da-1151-4843-891d-888c5057a133 is finished
2021-04-07 08:13:35,411 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6c9f90b5-3526-4c8b-9474-e20d0502c239, queue=awx_private_queue)
[2021-04-07 08:13:35,411: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6c9f90b5-3526-4c8b-9474-e20d0502c239, queue=awx_private_queue)
2021-04-07 08:13:35,434 DEBUG awx.main.dispatch delivered 6c9f90b5-3526-4c8b-9474-e20d0502c239 to worker[180] qsize 0
2021-04-07 08:13:35,441 DEBUG awx.main.dispatch task 6c9f90b5-3526-4c8b-9474-e20d0502c239 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:13:45,469 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a576147a-3d87-4161-8fd0-499b53a722fd, queue=awx_private_queue)
[2021-04-07 08:13:45,469: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a576147a-3d87-4161-8fd0-499b53a722fd, queue=awx_private_queue)
2021-04-07 08:13:45,497 DEBUG awx.main.dispatch delivered a576147a-3d87-4161-8fd0-499b53a722fd to worker[182] qsize 0
2021-04-07 08:13:45,505 DEBUG awx.main.dispatch task a576147a-3d87-4161-8fd0-499b53a722fd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:13:45,513 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:13:54,045 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(71ceb4b2-ac5a-4dea-959e-e2b32fc18b0c, queue=dev-awx-0)
[2021-04-07 08:13:54,045: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(71ceb4b2-ac5a-4dea-959e-e2b32fc18b0c, queue=dev-awx-0)
RESULT 2
OKREADY
2021-04-07 08:14:05,137 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8a9d72c4-589d-4ada-94de-5b184edab104, queue=dev-awx-0)
[2021-04-07 08:14:05,137: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8a9d72c4-589d-4ada-94de-5b184edab104, queue=dev-awx-0)
2021-04-07 08:14:05,409 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a0e60ee7-2027-4ffe-ad9c-7da23754ad88, queue=awx_private_queue)
[2021-04-07 08:14:05,409: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a0e60ee7-2027-4ffe-ad9c-7da23754ad88, queue=awx_private_queue)
2021-04-07 08:14:05,466 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b5948319-f7ee-4a3a-8fac-d095470d53be, queue=awx_private_queue)
[2021-04-07 08:14:05,466: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b5948319-f7ee-4a3a-8fac-d095470d53be, queue=awx_private_queue)
2021-04-07 08:14:19,003 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:14:25,485 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e88dd230-f599-40e7-b775-8094aa9beb63, queue=awx_private_queue)
[2021-04-07 08:14:25,485: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e88dd230-f599-40e7-b775-8094aa9beb63, queue=awx_private_queue)
RESULT 2
OKREADY
2021-04-07 08:15:11,183 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:15:11,329 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:13:05.475623+00:00
2021-04-07 08:15:11,242 DEBUG awx.main.dispatch delivered 71ceb4b2-ac5a-4dea-959e-e2b32fc18b0c to worker[183] qsize 0
2021-04-07 08:15:11,460 DEBUG awx.main.dispatch delivered 8a9d72c4-589d-4ada-94de-5b184edab104 to worker[181] qsize 0
2021-04-07 08:15:11,463 DEBUG awx.main.dispatch task 71ceb4b2-ac5a-4dea-959e-e2b32fc18b0c starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:15:11,474 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:15:11,466 DEBUG awx.main.dispatch task 8a9d72c4-589d-4ada-94de-5b184edab104 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:15:11,488 WARNING awx.main.dispatch scaling up worker pid:192
2021-04-07 08:15:11,494 DEBUG awx.main.dispatch delivered a0e60ee7-2027-4ffe-ad9c-7da23754ad88 to worker[192] qsize 0
2021-04-07 08:15:11,500 DEBUG awx.main.dispatch task a0e60ee7-2027-4ffe-ad9c-7da23754ad88 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:15:11,507 WARNING awx.main.dispatch scaling up worker pid:194
2021-04-07 08:15:11,512 DEBUG awx.main.dispatch delivered b5948319-f7ee-4a3a-8fac-d095470d53be to worker[194] qsize 0
2021-04-07 08:15:11,520 DEBUG awx.main.dispatch task b5948319-f7ee-4a3a-8fac-d095470d53be starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:15:11,522 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:15:11,523 WARNING awx.main.dispatch scaling up worker pid:196
2021-04-07 08:15:11,528 DEBUG awx.main.dispatch delivered e88dd230-f599-40e7-b775-8094aa9beb63 to worker[196] qsize 0
2021-04-07 08:15:11,534 DEBUG awx.main.dispatch task e88dd230-f599-40e7-b775-8094aa9beb63 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:15:11,536 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:14:35,409 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(39035c53-da9e-4583-8907-b653c20c0997, queue=awx_private_queue)
[2021-04-07 08:14:35,409: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(39035c53-da9e-4583-8907-b653c20c0997, queue=awx_private_queue)
2021-04-07 08:15:12,797 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8c736899-0e62-4417-ba51-01aa741cbc3e, queue=awx_private_queue)
[2021-04-07 08:15:12,797: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8c736899-0e62-4417-ba51-01aa741cbc3e, queue=awx_private_queue)
2021-04-07 08:15:12,805 WARNING awx.main.dispatch scaling up worker pid:198
2021-04-07 08:15:12,813 DEBUG awx.main.dispatch delivered 39035c53-da9e-4583-8907-b653c20c0997 to worker[198] qsize 0
2021-04-07 08:15:12,817 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(9580e748-cc0b-41ab-83e8-412fb156e640, queue=dev-awx-0)
[2021-04-07 08:15:12,817: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(9580e748-cc0b-41ab-83e8-412fb156e640, queue=dev-awx-0)
2021-04-07 08:15:12,823 DEBUG awx.main.dispatch task 39035c53-da9e-4583-8907-b653c20c0997 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:15:12,828 WARNING awx.main.dispatch scaling up worker pid:200
2021-04-07 08:15:12,834 DEBUG awx.main.dispatch delivered 8c736899-0e62-4417-ba51-01aa741cbc3e to worker[200] qsize 0
2021-04-07 08:15:12,837 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(09f1c2b0-0b34-44e6-bb31-e7063ad9a82e, queue=dev-awx-0)
[2021-04-07 08:15:12,837: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(09f1c2b0-0b34-44e6-bb31-e7063ad9a82e, queue=dev-awx-0)
2021-04-07 08:15:12,839 DEBUG awx.main.dispatch task 8c736899-0e62-4417-ba51-01aa741cbc3e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:15:12,841 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:15:32,808 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(37eb88be-d5bd-4ca5-9139-cf9e3afb4357, queue=awx_private_queue)
[2021-04-07 08:15:32,808: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(37eb88be-d5bd-4ca5-9139-cf9e3afb4357, queue=awx_private_queue)
2021-04-07 08:15:32,836 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4f81901f-4cc3-455f-a648-125a0810537c, queue=awx_private_queue)
[2021-04-07 08:15:32,836: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4f81901f-4cc3-455f-a648-125a0810537c, queue=awx_private_queue)
2021-04-07 08:15:45,699 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-07 08:15:45,700 DEBUG awx.main.dispatch task e88dd230-f599-40e7-b775-8094aa9beb63 is finished
2021-04-07 08:15:45,947 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-04-07 08:15:45,947 DEBUG awx.main.dispatch task a0e60ee7-2027-4ffe-ad9c-7da23754ad88 is finished
2021-04-07 08:15:52,821 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(050388f9-b579-4441-a3bb-f4796bbe9062, queue=awx_private_queue)
[2021-04-07 08:15:52,821: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(050388f9-b579-4441-a3bb-f4796bbe9062, queue=awx_private_queue)
2021-04-07 08:15:53,948 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(7687b878-68e1-49fe-814a-927ebb885600, queue=dev-awx-0)
[2021-04-07 08:15:53,948: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(7687b878-68e1-49fe-814a-927ebb885600, queue=dev-awx-0)
RESULT 2
OKREADY
2021-04-07 08:16:02,837 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4e3e2ded-3312-4f1a-bd03-0d0f6ce55a72, queue=awx_private_queue)
[2021-04-07 08:16:02,837: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4e3e2ded-3312-4f1a-bd03-0d0f6ce55a72, queue=awx_private_queue)
2021-04-07 08:16:56,601 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-07 08:16:12,817 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e265be69-de43-4180-b6bb-0c49375d1be4, queue=dev-awx-0)
[2021-04-07 08:16:12,817: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e265be69-de43-4180-b6bb-0c49375d1be4, queue=dev-awx-0)
2021-04-07 08:16:56,800 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3165858d-df17-4643-85c6-d944bb99ef1b, queue=awx_private_queue)
[2021-04-07 08:16:56,800: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3165858d-df17-4643-85c6-d944bb99ef1b, queue=awx_private_queue)
2021-04-07 08:16:56,813 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(0e04a53b-037e-4267-9e0d-1272cb809185, queue=dev-awx-0)
[2021-04-07 08:16:56,813: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(0e04a53b-037e-4267-9e0d-1272cb809185, queue=dev-awx-0)
2021-04-07 08:16:56,564 DEBUG awx.main.dispatch task 8a9d72c4-589d-4ada-94de-5b184edab104 is finished
2021-04-07 08:16:56,826 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dd74d0f3-3565-4172-8f17-140b870b3181, queue=awx_private_queue)
[2021-04-07 08:16:56,826: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dd74d0f3-3565-4172-8f17-140b870b3181, queue=awx_private_queue)
2021-04-07 08:16:56,603 DEBUG awx.main.dispatch task b5948319-f7ee-4a3a-8fac-d095470d53be is finished
2021-04-07 08:16:57,819 DEBUG awx.main.dispatch task 6c9f90b5-3526-4c8b-9474-e20d0502c239 is finished
2021-04-07 08:16:57,840 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:16:57,980 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:14:19.004531+00:00
2021-04-07 08:16:58,043 DEBUG awx.main.dispatch task 39035c53-da9e-4583-8907-b653c20c0997 is finished
RESULT 2
OKREADY
2021-04-07 08:17:12,427 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-07 08:17:12,428 DEBUG awx.main.dispatch task 8c736899-0e62-4417-ba51-01aa741cbc3e is finished
2021-04-07 08:17:12,469 DEBUG awx.main.dispatch delivered 9580e748-cc0b-41ab-83e8-412fb156e640 to worker[196] qsize 0
2021-04-07 08:17:12,478 DEBUG awx.main.dispatch delivered 09f1c2b0-0b34-44e6-bb31-e7063ad9a82e to worker[180] qsize 0
2021-04-07 08:17:12,480 DEBUG awx.main.dispatch delivered 37eb88be-d5bd-4ca5-9139-cf9e3afb4357 to worker[198] qsize 0
2021-04-07 08:17:12,483 DEBUG awx.main.dispatch delivered 4f81901f-4cc3-455f-a648-125a0810537c to worker[192] qsize 0
2021-04-07 08:17:12,481 DEBUG awx.main.dispatch task 09f1c2b0-0b34-44e6-bb31-e7063ad9a82e starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:17:12,484 DEBUG awx.main.dispatch delivered 050388f9-b579-4441-a3bb-f4796bbe9062 to worker[181] qsize 0
2021-04-07 08:17:12,480 DEBUG awx.main.dispatch task 9580e748-cc0b-41ab-83e8-412fb156e640 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:17:12,489 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:17:12,490 DEBUG awx.main.dispatch delivered 7687b878-68e1-49fe-814a-927ebb885600 to worker[200] qsize 0
2021-04-07 08:17:12,488 DEBUG awx.main.dispatch task 37eb88be-d5bd-4ca5-9139-cf9e3afb4357 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:17:12,492 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:17:12,492 DEBUG awx.main.dispatch delivered 4e3e2ded-3312-4f1a-bd03-0d0f6ce55a72 to worker[194] qsize 0
2021-04-07 08:17:12,493 DEBUG awx.main.dispatch task 7687b878-68e1-49fe-814a-927ebb885600 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-04-07 08:17:12,496 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-04-07 08:17:12,493 DEBUG awx.main.dispatch task 050388f9-b579-4441-a3bb-f4796bbe9062 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:17:12,502 DEBUG awx.main.dispatch task 4e3e2ded-3312-4f1a-bd03-0d0f6ce55a72 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:17:12,506 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:17:12,499 DEBUG awx.main.dispatch task 4f81901f-4cc3-455f-a648-125a0810537c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:17:12,531 WARNING awx.main.tasks Rejoining the cluster as instance dev-awx-0.
2021-04-07 08:17:12,533 DEBUG awx.main.dispatch task 71ceb4b2-ac5a-4dea-959e-e2b32fc18b0c is finished
2021-04-07 08:17:12,553 WARNING awx.main.dispatch scaling up worker pid:207
2021-04-07 08:17:12,560 DEBUG awx.main.dispatch delivered e265be69-de43-4180-b6bb-0c49375d1be4 to worker[183] qsize 0
2021-04-07 08:17:12,564 DEBUG awx.main.dispatch delivered 3165858d-df17-4643-85c6-d944bb99ef1b to worker[207] qsize 0
2021-04-07 08:17:12,564 DEBUG awx.main.dispatch task e265be69-de43-4180-b6bb-0c49375d1be4 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:17:12,566 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:17:12,576 WARNING awx.main.dispatch scaling up worker pid:209
2021-04-07 08:17:12,581 DEBUG awx.main.dispatch delivered 0e04a53b-037e-4267-9e0d-1272cb809185 to worker[209] qsize 0
2021-04-07 08:17:12,573 DEBUG awx.main.dispatch task 3165858d-df17-4643-85c6-d944bb99ef1b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:17:12,583 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:17:12,585 DEBUG awx.main.dispatch task 0e04a53b-037e-4267-9e0d-1272cb809185 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:17:12,593 WARNING awx.main.dispatch scaling up worker pid:211
2021-04-07 08:17:12,599 DEBUG awx.main.dispatch delivered dd74d0f3-3565-4172-8f17-140b870b3181 to worker[211] qsize 0
2021-04-07 08:17:12,605 DEBUG awx.main.dispatch task dd74d0f3-3565-4172-8f17-140b870b3181 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:17:16,801 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2c54b4aa-5920-4d40-b032-4b8c99170818, queue=awx_private_queue)
[2021-04-07 08:17:16,801: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2c54b4aa-5920-4d40-b032-4b8c99170818, queue=awx_private_queue)
2021-04-07 08:17:16,832 WARNING awx.main.dispatch scaling up worker pid:213
2021-04-07 08:17:16,839 DEBUG awx.main.dispatch delivered 2c54b4aa-5920-4d40-b032-4b8c99170818 to worker[213] qsize 0
2021-04-07 08:17:16,847 DEBUG awx.main.dispatch task 2c54b4aa-5920-4d40-b032-4b8c99170818 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:17:16,852 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:17:26,827 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(05effe54-d96c-46d3-a7e0-867b2518a67c, queue=awx_private_queue)
[2021-04-07 08:17:26,827: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(05effe54-d96c-46d3-a7e0-867b2518a67c, queue=awx_private_queue)
2021-04-07 08:17:26,900 WARNING awx.main.dispatch scaling up worker pid:215
2021-04-07 08:17:26,907 DEBUG awx.main.dispatch delivered 05effe54-d96c-46d3-a7e0-867b2518a67c to worker[215] qsize 0
2021-04-07 08:17:26,912 DEBUG awx.main.dispatch task 05effe54-d96c-46d3-a7e0-867b2518a67c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:17:36,800 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(950f2a2f-cf67-49ce-8d85-7d502dc5bb0b, queue=awx_private_queue)
[2021-04-07 08:17:36,800: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(950f2a2f-cf67-49ce-8d85-7d502dc5bb0b, queue=awx_private_queue)
2021-04-07 08:17:36,829 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(1ea7d55a-2e15-4452-baed-2ae4d5a2e6ca, queue=dev-awx-0)
[2021-04-07 08:17:36,829: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(1ea7d55a-2e15-4452-baed-2ae4d5a2e6ca, queue=dev-awx-0)
2021-04-07 08:17:36,842 WARNING awx.main.dispatch scaling up worker pid:217
2021-04-07 08:17:36,850 DEBUG awx.main.dispatch delivered 950f2a2f-cf67-49ce-8d85-7d502dc5bb0b to worker[217] qsize 0
2021-04-07 08:17:36,856 DEBUG awx.main.dispatch task 950f2a2f-cf67-49ce-8d85-7d502dc5bb0b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:17:36,860 DEBUG awx.main.scheduler Running Tower task manager.
RESULT 2
OKREADY
2021-04-07 08:18:30,728 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:18:33,544 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-04-07 08:18:33,840 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-07 08:18:33,842 DEBUG awx.main.dispatch task 050388f9-b579-4441-a3bb-f4796bbe9062 is finished
2021-04-07 08:18:30,902 DEBUG awx.main.dispatch task 09f1c2b0-0b34-44e6-bb31-e7063ad9a82e is finished
2021-04-07 08:18:33,548 DEBUG awx.main.dispatch task 4f81901f-4cc3-455f-a648-125a0810537c is finished
2021-04-07 08:18:42,194 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:16:57.841944+00:00
RESULT 2
OKREADY
2021-04-07 08:19:05,418 DEBUG awx.main.dispatch task 0e04a53b-037e-4267-9e0d-1272cb809185 is finished
2021-04-07 08:19:07,217 DEBUG awx.main.dispatch task e265be69-de43-4180-b6bb-0c49375d1be4 is finished
2021-04-07 08:19:07,228 DEBUG awx.main.dispatch task 9580e748-cc0b-41ab-83e8-412fb156e640 is finished
2021-04-07 08:19:07,711 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-04-07 08:19:07,714 DEBUG awx.main.dispatch task 4e3e2ded-3312-4f1a-bd03-0d0f6ce55a72 is finished
2021-04-07 08:19:09,405 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-07 08:19:09,406 DEBUG awx.main.dispatch task 950f2a2f-cf67-49ce-8d85-7d502dc5bb0b is finished
2021-04-07 08:19:09,717 DEBUG awx.main.dispatch task a576147a-3d87-4161-8fd0-499b53a722fd is finished
2021-04-07 08:19:39,871 DEBUG awx.main.dispatch task 7687b878-68e1-49fe-814a-927ebb885600 is finished
2021-04-07 08:19:46,069 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:19:47,803 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-07 08:19:47,804 DEBUG awx.main.dispatch task 2c54b4aa-5920-4d40-b032-4b8c99170818 is finished
2021-04-07 08:19:48,347 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-04-07 08:19:48,349 DEBUG awx.main.dispatch task 05effe54-d96c-46d3-a7e0-867b2518a67c is finished
2021-04-07 08:17:56,828 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c3dcef1f-45ac-4619-b74b-1935d1404f6f, queue=dev-awx-0)
[2021-04-07 08:17:56,828: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c3dcef1f-45ac-4619-b74b-1935d1404f6f, queue=dev-awx-0)
2021-04-07 08:19:48,639 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1ec2339b-d614-444f-a412-e0b1e0ed98f2, queue=awx_private_queue)
[2021-04-07 08:19:48,639: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1ec2339b-d614-444f-a412-e0b1e0ed98f2, queue=awx_private_queue)
2021-04-07 08:19:48,657 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9b7a9def-54fb-4e5b-a5ed-42ef2ef0326f, queue=awx_private_queue)
[2021-04-07 08:19:48,657: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9b7a9def-54fb-4e5b-a5ed-42ef2ef0326f, queue=awx_private_queue)
2021-04-07 08:19:48,674 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(18a8072a-406c-4d15-aa73-81c620de2a0a, queue=dev-awx-0)
[2021-04-07 08:19:48,674: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(18a8072a-406c-4d15-aa73-81c620de2a0a, queue=dev-awx-0)
2021-04-07 08:19:50,503 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-07 08:19:50,503 DEBUG awx.main.dispatch task 3165858d-df17-4643-85c6-d944bb99ef1b is finished
2021-04-07 08:19:52,393 DEBUG awx.main.dispatch delivered 1ea7d55a-2e15-4452-baed-2ae4d5a2e6ca to worker[180] qsize 0
2021-04-07 08:19:52,405 DEBUG awx.main.dispatch delivered c3dcef1f-45ac-4619-b74b-1935d1404f6f to worker[182] qsize 0
2021-04-07 08:19:52,407 DEBUG awx.main.dispatch delivered 1ec2339b-d614-444f-a412-e0b1e0ed98f2 to worker[183] qsize 0
2021-04-07 08:19:52,409 DEBUG awx.main.dispatch task c3dcef1f-45ac-4619-b74b-1935d1404f6f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:19:52,414 DEBUG awx.main.dispatch task 1ec2339b-d614-444f-a412-e0b1e0ed98f2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:19:52,420 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:19:52,406 DEBUG awx.main.dispatch task 1ea7d55a-2e15-4452-baed-2ae4d5a2e6ca starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:19:52,420 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:19:52,425 DEBUG awx.main.dispatch delivered 9b7a9def-54fb-4e5b-a5ed-42ef2ef0326f to worker[181] qsize 0
2021-04-07 08:19:52,427 WARNING awx.main.dispatch scaling down worker pid:192
2021-04-07 08:19:52,428 DEBUG awx.main.dispatch task 9b7a9def-54fb-4e5b-a5ed-42ef2ef0326f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:19:52,431 WARNING awx.main.dispatch scaling down worker pid:194
2021-04-07 08:19:52,434 WARNING awx.main.dispatch scaling down worker pid:196
2021-04-07 08:19:52,436 WARNING awx.main.dispatch scaling down worker pid:200
2021-04-07 08:19:52,437 WARNING awx.main.dispatch scaling down worker pid:207
2021-04-07 08:19:52,439 WARNING awx.main.dispatch scaling down worker pid:209
2021-04-07 08:19:52,435 WARNING awx.main.dispatch worker exiting gracefully pid:194
2021-04-07 08:19:52,440 WARNING awx.main.dispatch worker exiting gracefully pid:207
2021-04-07 08:19:52,437 WARNING awx.main.dispatch worker exiting gracefully pid:196
2021-04-07 08:19:52,439 WARNING awx.main.dispatch worker exiting gracefully pid:200
2021-04-07 08:19:52,433 WARNING awx.main.dispatch worker exiting gracefully pid:192
2021-04-07 08:19:52,441 WARNING awx.main.dispatch scaling down worker pid:213
2021-04-07 08:19:52,442 WARNING awx.main.dispatch worker exiting gracefully pid:209
2021-04-07 08:19:52,451 WARNING awx.main.dispatch scaling down worker pid:215
2021-04-07 08:19:52,463 WARNING awx.main.dispatch worker exiting gracefully pid:213
2021-04-07 08:19:52,465 WARNING awx.main.dispatch scaling down worker pid:217
2021-04-07 08:19:52,467 WARNING awx.main.dispatch worker exiting gracefully pid:215
2021-04-07 08:19:52,472 WARNING awx.main.dispatch worker exiting gracefully pid:217
2021-04-07 08:19:54,514 WARNING awx.main.dispatch scaling up worker pid:224
2021-04-07 08:19:54,521 DEBUG awx.main.dispatch delivered 18a8072a-406c-4d15-aa73-81c620de2a0a to worker[224] qsize 0
2021-04-07 08:19:54,526 DEBUG awx.main.dispatch task 18a8072a-406c-4d15-aa73-81c620de2a0a starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-07 08:19:54,530 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-07 08:19:55,376 DEBUG awx.main.dispatch task dd74d0f3-3565-4172-8f17-140b870b3181 is finished
2021-04-07 08:19:56,244 DEBUG awx.main.dispatch task 37eb88be-d5bd-4ca5-9139-cf9e3afb4357 is finished
RESULT 2
OKREADY
2021-04-07 08:20:08,649 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b8d1b0ee-a8af-47eb-a915-44907182145b, queue=awx_private_queue)
[2021-04-07 08:20:08,649: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b8d1b0ee-a8af-47eb-a915-44907182145b, queue=awx_private_queue)
2021-04-07 08:20:08,684 DEBUG awx.main.dispatch delivered b8d1b0ee-a8af-47eb-a915-44907182145b to worker[198] qsize 0
2021-04-07 08:20:08,687 DEBUG awx.main.dispatch task b8d1b0ee-a8af-47eb-a915-44907182145b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:20:08,688 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:20:08,697 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:20:08,706 DEBUG awx.main.dispatch task b8d1b0ee-a8af-47eb-a915-44907182145b is finished
2021-04-07 08:20:18,663 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(55b18707-b6a3-473e-8912-61c88320e40f, queue=awx_private_queue)
[2021-04-07 08:20:18,663: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(55b18707-b6a3-473e-8912-61c88320e40f, queue=awx_private_queue)
2021-04-07 08:20:18,701 DEBUG awx.main.dispatch delivered 55b18707-b6a3-473e-8912-61c88320e40f to worker[211] qsize 0
2021-04-07 08:20:18,707 DEBUG awx.main.dispatch task 55b18707-b6a3-473e-8912-61c88320e40f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:20:18,730 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:20:18,738 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:20:18,748 DEBUG awx.main.dispatch task 55b18707-b6a3-473e-8912-61c88320e40f is finished
2021-04-07 08:20:28,653 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f21775d3-f42f-46e6-afc5-27d57a510d2e, queue=awx_private_queue)
[2021-04-07 08:20:28,653: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f21775d3-f42f-46e6-afc5-27d57a510d2e, queue=awx_private_queue)
2021-04-07 08:20:28,682 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(adc243bd-735d-47bf-9bfc-9a3edbf3e361, queue=dev-awx-0)
[2021-04-07 08:20:28,682: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(adc243bd-735d-47bf-9bfc-9a3edbf3e361, queue=dev-awx-0)
2021-04-07 08:20:28,679 DEBUG awx.main.dispatch delivered f21775d3-f42f-46e6-afc5-27d57a510d2e to worker[211] qsize 0
2021-04-07 08:20:28,696 DEBUG awx.main.dispatch task f21775d3-f42f-46e6-afc5-27d57a510d2e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:20:28,698 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:20:28,703 DEBUG awx.main.dispatch delivered adc243bd-735d-47bf-9bfc-9a3edbf3e361 to worker[198] qsize 0
2021-04-07 08:20:28,708 DEBUG awx.main.dispatch task adc243bd-735d-47bf-9bfc-9a3edbf3e361 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:20:28,715 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:20:28,718 DEBUG awx.main.dispatch task f21775d3-f42f-46e6-afc5-27d57a510d2e is finished
2021-04-07 08:20:28,722 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139995040589376_1" does not exist
2021-04-07 08:20:28,725 DEBUG awx.main.dispatch task adc243bd-735d-47bf-9bfc-9a3edbf3e361 is finished
2021-04-07 08:20:48,663 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(25899677-37b8-4f71-a051-3930063df641, queue=awx_private_queue)
[2021-04-07 08:20:48,663: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(25899677-37b8-4f71-a051-3930063df641, queue=awx_private_queue)
2021-04-07 08:20:49,621 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f31a0b3f-89ea-43d9-b9f5-08ccefa0d2d0, queue=dev-awx-0)
[2021-04-07 08:20:49,621: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f31a0b3f-89ea-43d9-b9f5-08ccefa0d2d0, queue=dev-awx-0)
2021-04-07 08:20:49,653 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(aadea793-1d11-4f79-b30d-de969dc779fe, queue=awx_private_queue)
[2021-04-07 08:20:49,653: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(aadea793-1d11-4f79-b30d-de969dc779fe, queue=awx_private_queue)
2021-04-07 08:20:49,623 DEBUG awx.main.dispatch delivered 25899677-37b8-4f71-a051-3930063df641 to worker[198] qsize 0
2021-04-07 08:20:49,815 WARNING awx.main.dispatch scaling down worker pid:211
2021-04-07 08:20:49,833 WARNING awx.main.dispatch worker exiting gracefully pid:211
2021-04-07 08:20:49,822 DEBUG awx.main.dispatch task 25899677-37b8-4f71-a051-3930063df641 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:20:49,862 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:20:49,847 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-07 08:20:49,864 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:20:49,872 DEBUG awx.main.dispatch task 25899677-37b8-4f71-a051-3930063df641 is finished
2021-04-07 08:20:49,878 WARNING awx.main.dispatch scaling up worker pid:227
2021-04-07 08:20:49,943 DEBUG awx.main.dispatch delivered aadea793-1d11-4f79-b30d-de969dc779fe to worker[198] qsize 0
2021-04-07 08:20:49,960 DEBUG awx.main.dispatch task aadea793-1d11-4f79-b30d-de969dc779fe starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:20:49,986 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:20:49,999 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:20:50,001 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:20:50,008 DEBUG awx.main.dispatch task aadea793-1d11-4f79-b30d-de969dc779fe is finished
RESULT 2
OKREADY
2021-04-07 08:21:07,293 DEBUG awx.main.scheduler Starting Scheduler
2021-04-07 08:21:08,672 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c001a255-1962-4c18-a590-777be607bf79, queue=awx_private_queue)
[2021-04-07 08:21:08,672: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c001a255-1962-4c18-a590-777be607bf79, queue=awx_private_queue)
2021-04-07 08:21:08,725 DEBUG awx.main.dispatch delivered c001a255-1962-4c18-a590-777be607bf79 to worker[227] qsize 0
2021-04-07 08:21:08,763 DEBUG awx.main.dispatch task c001a255-1962-4c18-a590-777be607bf79 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:21:08,787 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:21:08,793 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:21:08,800 DEBUG awx.main.dispatch task c001a255-1962-4c18-a590-777be607bf79 is finished
2021-04-07 08:20:48,561 DEBUG awx.main.dispatch task c3dcef1f-45ac-4619-b74b-1935d1404f6f is finished
2021-04-07 08:21:07,196 DEBUG awx.main.dispatch task 1ea7d55a-2e15-4452-baed-2ae4d5a2e6ca is finished
2021-04-07 08:21:12,081 DEBUG awx.main.tasks Last scheduler run was: 2021-04-07 08:18:30.729829+00:00
2021-04-07 08:21:13,351 DEBUG awx.main.dispatch task 9b7a9def-54fb-4e5b-a5ed-42ef2ef0326f is finished
2021-04-07 08:21:16,747 DEBUG awx.main.dispatch task 1ec2339b-d614-444f-a412-e0b1e0ed98f2 is finished
2021-04-07 08:21:19,656 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(16780c45-33ae-46d2-aa5d-c77450fc76fc, queue=awx_private_queue)
[2021-04-07 08:21:19,656: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(16780c45-33ae-46d2-aa5d-c77450fc76fc, queue=awx_private_queue)
2021-04-07 08:21:19,678 DEBUG awx.main.dispatch delivered 16780c45-33ae-46d2-aa5d-c77450fc76fc to worker[198] qsize 0
2021-04-07 08:21:19,686 DEBUG awx.main.dispatch task 16780c45-33ae-46d2-aa5d-c77450fc76fc starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:21:19,702 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:21:19,706 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:21:19,709 DEBUG awx.main.dispatch task 16780c45-33ae-46d2-aa5d-c77450fc76fc is finished
2021-04-07 08:21:01,404 DEBUG awx.main.dispatch task 18a8072a-406c-4d15-aa73-81c620de2a0a is finished
2021-04-07 08:21:28,684 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(0a960ca6-7bea-4f5a-8845-4159480b5449, queue=dev-awx-0)
[2021-04-07 08:21:28,684: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(0a960ca6-7bea-4f5a-8845-4159480b5449, queue=dev-awx-0)
2021-04-07 08:21:28,707 DEBUG awx.main.dispatch delivered 0a960ca6-7bea-4f5a-8845-4159480b5449 to worker[224] qsize 0
2021-04-07 08:21:28,708 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fbe4ef86-31d3-4a69-a855-1fb73927df59, queue=awx_private_queue)
[2021-04-07 08:21:28,708: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fbe4ef86-31d3-4a69-a855-1fb73927df59, queue=awx_private_queue)
2021-04-07 08:21:28,710 DEBUG awx.main.dispatch task 0a960ca6-7bea-4f5a-8845-4159480b5449 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:21:28,721 DEBUG awx.main.dispatch delivered fbe4ef86-31d3-4a69-a855-1fb73927df59 to worker[183] qsize 0
2021-04-07 08:21:28,724 DEBUG awx.main.dispatch task fbe4ef86-31d3-4a69-a855-1fb73927df59 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:21:28,726 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:21:28,761 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139995040589376_1" does not exist
2021-04-07 08:21:28,753 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:21:28,819 DEBUG awx.main.dispatch task fbe4ef86-31d3-4a69-a855-1fb73927df59 is finished
2021-04-07 08:21:28,818 DEBUG awx.main.dispatch task 0a960ca6-7bea-4f5a-8845-4159480b5449 is finished
2021-04-07 08:21:48,719 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4e7d75cd-d2c0-4257-b18a-b15801f6b171, queue=awx_private_queue)
[2021-04-07 08:21:48,719: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4e7d75cd-d2c0-4257-b18a-b15801f6b171, queue=awx_private_queue)
2021-04-07 08:21:48,745 DEBUG awx.main.dispatch delivered 4e7d75cd-d2c0-4257-b18a-b15801f6b171 to worker[227] qsize 0
2021-04-07 08:21:48,755 DEBUG awx.main.dispatch task 4e7d75cd-d2c0-4257-b18a-b15801f6b171 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:21:48,758 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:21:48,881 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:21:48,884 DEBUG awx.main.dispatch task 4e7d75cd-d2c0-4257-b18a-b15801f6b171 is finished
2021-04-07 08:21:49,620 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5c0c79ee-f6c8-46bf-94ca-88545dc7d552, queue=dev-awx-0)
[2021-04-07 08:21:49,620: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5c0c79ee-f6c8-46bf-94ca-88545dc7d552, queue=dev-awx-0)
2021-04-07 08:21:49,634 WARNING awx.main.dispatch scaling down worker pid:180
2021-04-07 08:21:49,638 WARNING awx.main.dispatch scaling down worker pid:181
2021-04-07 08:21:49,640 WARNING awx.main.dispatch scaling down worker pid:182
2021-04-07 08:21:49,640 WARNING awx.main.dispatch worker exiting gracefully pid:180
2021-04-07 08:21:49,643 WARNING awx.main.dispatch worker exiting gracefully pid:182
2021-04-07 08:21:49,650 WARNING awx.main.dispatch worker exiting gracefully pid:181
2021-04-07 08:21:49,653 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1172651e-5a2a-47ec-84a9-8fd74fb79fd8, queue=awx_private_queue)
[2021-04-07 08:21:49,653: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1172651e-5a2a-47ec-84a9-8fd74fb79fd8, queue=awx_private_queue)
2021-04-07 08:21:49,719 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-07 08:21:49,722 DEBUG awx.main.dispatch delivered 1172651e-5a2a-47ec-84a9-8fd74fb79fd8 to worker[227] qsize 0
2021-04-07 08:21:49,730 DEBUG awx.main.dispatch task 1172651e-5a2a-47ec-84a9-8fd74fb79fd8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:21:49,767 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:21:49,796 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:21:49,811 DEBUG awx.main.dispatch task 1172651e-5a2a-47ec-84a9-8fd74fb79fd8 is finished
RESULT 2
OKREADY
2021-04-07 08:22:08,732 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cd1e6155-7e9e-4521-9ea5-969a239d71f3, queue=awx_private_queue)
[2021-04-07 08:22:08,732: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cd1e6155-7e9e-4521-9ea5-969a239d71f3, queue=awx_private_queue)
2021-04-07 08:22:08,760 DEBUG awx.main.dispatch delivered cd1e6155-7e9e-4521-9ea5-969a239d71f3 to worker[183] qsize 0
2021-04-07 08:22:08,775 DEBUG awx.main.dispatch task cd1e6155-7e9e-4521-9ea5-969a239d71f3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:22:08,779 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:22:08,806 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:22:08,809 DEBUG awx.main.dispatch task cd1e6155-7e9e-4521-9ea5-969a239d71f3 is finished
2021-04-07 08:22:19,653 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(92a7ddbb-0430-4edb-b47a-f5332001a3e2, queue=awx_private_queue)
[2021-04-07 08:22:19,653: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(92a7ddbb-0430-4edb-b47a-f5332001a3e2, queue=awx_private_queue)
2021-04-07 08:22:19,669 DEBUG awx.main.dispatch delivered 92a7ddbb-0430-4edb-b47a-f5332001a3e2 to worker[227] qsize 0
2021-04-07 08:22:19,678 DEBUG awx.main.dispatch task 92a7ddbb-0430-4edb-b47a-f5332001a3e2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:22:19,729 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:22:19,751 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:22:19,755 DEBUG awx.main.dispatch task 92a7ddbb-0430-4edb-b47a-f5332001a3e2 is finished
2021-04-07 08:22:28,684 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5c119895-0f3a-4e89-84c0-a2a9d004ed1c, queue=dev-awx-0)
[2021-04-07 08:22:28,684: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5c119895-0f3a-4e89-84c0-a2a9d004ed1c, queue=dev-awx-0)
2021-04-07 08:22:28,744 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(faa2b144-1ba0-4332-bbbe-528dd02dc8c6, queue=awx_private_queue)
[2021-04-07 08:22:28,744: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(faa2b144-1ba0-4332-bbbe-528dd02dc8c6, queue=awx_private_queue)
2021-04-07 08:22:28,737 DEBUG awx.main.dispatch delivered 5c119895-0f3a-4e89-84c0-a2a9d004ed1c to worker[227] qsize 0
2021-04-07 08:22:28,783 DEBUG awx.main.dispatch delivered faa2b144-1ba0-4332-bbbe-528dd02dc8c6 to worker[198] qsize 0
2021-04-07 08:22:28,772 DEBUG awx.main.dispatch task 5c119895-0f3a-4e89-84c0-a2a9d004ed1c starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:22:28,831 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139995040589376_1" does not exist
2021-04-07 08:22:28,804 DEBUG awx.main.dispatch task faa2b144-1ba0-4332-bbbe-528dd02dc8c6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:22:28,840 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:22:28,837 DEBUG awx.main.dispatch task 5c119895-0f3a-4e89-84c0-a2a9d004ed1c is finished
2021-04-07 08:22:28,842 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:22:28,848 DEBUG awx.main.dispatch task faa2b144-1ba0-4332-bbbe-528dd02dc8c6 is finished
2021-04-07 08:22:48,751 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ebf27664-d221-4163-8276-0aedae9ca5c2, queue=awx_private_queue)
[2021-04-07 08:22:48,751: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ebf27664-d221-4163-8276-0aedae9ca5c2, queue=awx_private_queue)
2021-04-07 08:22:48,787 DEBUG awx.main.dispatch delivered ebf27664-d221-4163-8276-0aedae9ca5c2 to worker[198] qsize 0
2021-04-07 08:22:48,800 DEBUG awx.main.dispatch task ebf27664-d221-4163-8276-0aedae9ca5c2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:22:48,811 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:22:48,812 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:22:48,816 DEBUG awx.main.dispatch task ebf27664-d221-4163-8276-0aedae9ca5c2 is finished
2021-04-07 08:22:49,622 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(12f0674d-b572-436d-81f5-34a2d784d68a, queue=dev-awx-0)
[2021-04-07 08:22:49,622: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(12f0674d-b572-436d-81f5-34a2d784d68a, queue=dev-awx-0)
2021-04-07 08:22:49,653 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(aee20d05-c560-4904-8398-0e44d451a1b1, queue=awx_private_queue)
[2021-04-07 08:22:49,653: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(aee20d05-c560-4904-8398-0e44d451a1b1, queue=awx_private_queue)
2021-04-07 08:22:49,656 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-07 08:22:49,676 DEBUG awx.main.dispatch delivered aee20d05-c560-4904-8398-0e44d451a1b1 to worker[227] qsize 0
2021-04-07 08:22:49,685 DEBUG awx.main.dispatch task aee20d05-c560-4904-8398-0e44d451a1b1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:22:49,702 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:22:49,705 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:22:49,711 DEBUG awx.main.dispatch task aee20d05-c560-4904-8398-0e44d451a1b1 is finished
RESULT 2
OKREADY
2021-04-07 08:23:08,770 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(458e36b5-9ddf-437c-a48c-ff7bf3d42ff5, queue=awx_private_queue)
[2021-04-07 08:23:08,770: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(458e36b5-9ddf-437c-a48c-ff7bf3d42ff5, queue=awx_private_queue)
2021-04-07 08:23:08,835 DEBUG awx.main.dispatch delivered 458e36b5-9ddf-437c-a48c-ff7bf3d42ff5 to worker[227] qsize 0
2021-04-07 08:23:08,855 DEBUG awx.main.dispatch task 458e36b5-9ddf-437c-a48c-ff7bf3d42ff5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:23:08,895 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:23:08,901 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:23:08,906 DEBUG awx.main.dispatch task 458e36b5-9ddf-437c-a48c-ff7bf3d42ff5 is finished
2021-04-07 08:23:19,658 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ec05e652-c59c-4741-b03b-22457745cee1, queue=awx_private_queue)
[2021-04-07 08:23:19,658: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ec05e652-c59c-4741-b03b-22457745cee1, queue=awx_private_queue)
2021-04-07 08:23:19,698 DEBUG awx.main.dispatch delivered ec05e652-c59c-4741-b03b-22457745cee1 to worker[227] qsize 0
2021-04-07 08:23:19,713 DEBUG awx.main.dispatch task ec05e652-c59c-4741-b03b-22457745cee1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:23:19,737 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:23:19,741 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:23:19,746 DEBUG awx.main.dispatch task ec05e652-c59c-4741-b03b-22457745cee1 is finished
2021-04-07 08:23:28,687 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f78f395b-cfd1-4050-a9a9-5fe14b6893a8, queue=dev-awx-0)
[2021-04-07 08:23:28,687: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f78f395b-cfd1-4050-a9a9-5fe14b6893a8, queue=dev-awx-0)
2021-04-07 08:23:28,716 DEBUG awx.main.dispatch delivered f78f395b-cfd1-4050-a9a9-5fe14b6893a8 to worker[224] qsize 0
2021-04-07 08:23:28,759 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9a4d4d07-fe8b-4b91-8aeb-2f2398940f9b, queue=awx_private_queue)
[2021-04-07 08:23:28,759: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9a4d4d07-fe8b-4b91-8aeb-2f2398940f9b, queue=awx_private_queue)
2021-04-07 08:23:28,782 DEBUG awx.main.dispatch delivered 9a4d4d07-fe8b-4b91-8aeb-2f2398940f9b to worker[198] qsize 0
2021-04-07 08:23:28,798 DEBUG awx.main.dispatch task 9a4d4d07-fe8b-4b91-8aeb-2f2398940f9b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:23:28,828 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:23:28,832 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:23:28,844 DEBUG awx.main.dispatch task 9a4d4d07-fe8b-4b91-8aeb-2f2398940f9b is finished
2021-04-07 08:23:28,730 DEBUG awx.main.dispatch task f78f395b-cfd1-4050-a9a9-5fe14b6893a8 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:23:28,932 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139995040589376_2" does not exist
2021-04-07 08:23:28,942 DEBUG awx.main.dispatch task f78f395b-cfd1-4050-a9a9-5fe14b6893a8 is finished
2021-04-07 08:23:48,775 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9803dc60-fc25-480d-87e6-0a12f613ddfc, queue=awx_private_queue)
[2021-04-07 08:23:48,775: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9803dc60-fc25-480d-87e6-0a12f613ddfc, queue=awx_private_queue)
2021-04-07 08:23:48,800 DEBUG awx.main.dispatch delivered 9803dc60-fc25-480d-87e6-0a12f613ddfc to worker[183] qsize 0
2021-04-07 08:23:48,812 DEBUG awx.main.dispatch task 9803dc60-fc25-480d-87e6-0a12f613ddfc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:23:48,839 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:23:48,848 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:23:48,856 DEBUG awx.main.dispatch task 9803dc60-fc25-480d-87e6-0a12f613ddfc is finished
2021-04-07 08:23:49,621 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(1a6a0e5a-df2e-43cc-927f-84f84a958c2a, queue=dev-awx-0)
[2021-04-07 08:23:49,621: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(1a6a0e5a-df2e-43cc-927f-84f84a958c2a, queue=dev-awx-0)
2021-04-07 08:23:49,655 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4d727f73-96aa-4cad-b4ad-be00ffc4f3c1, queue=awx_private_queue)
[2021-04-07 08:23:49,655: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4d727f73-96aa-4cad-b4ad-be00ffc4f3c1, queue=awx_private_queue)
2021-04-07 08:23:49,659 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-07 08:23:49,677 DEBUG awx.main.dispatch delivered 4d727f73-96aa-4cad-b4ad-be00ffc4f3c1 to worker[227] qsize 0
2021-04-07 08:23:49,684 DEBUG awx.main.dispatch task 4d727f73-96aa-4cad-b4ad-be00ffc4f3c1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:23:49,701 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:23:49,704 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:23:49,709 DEBUG awx.main.dispatch task 4d727f73-96aa-4cad-b4ad-be00ffc4f3c1 is finished
RESULT 2
OKREADY
2021-04-07 08:24:08,789 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(df29ab93-6ee3-45ac-b4b4-c4325003dd16, queue=awx_private_queue)
[2021-04-07 08:24:08,789: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(df29ab93-6ee3-45ac-b4b4-c4325003dd16, queue=awx_private_queue)
2021-04-07 08:24:08,837 DEBUG awx.main.dispatch delivered df29ab93-6ee3-45ac-b4b4-c4325003dd16 to worker[224] qsize 0
2021-04-07 08:24:08,883 DEBUG awx.main.dispatch task df29ab93-6ee3-45ac-b4b4-c4325003dd16 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:24:08,898 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:24:08,900 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:24:08,911 DEBUG awx.main.dispatch task df29ab93-6ee3-45ac-b4b4-c4325003dd16 is finished
2021-04-07 08:24:19,659 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6e1ae650-886d-4e8d-9d2f-d13565756ad5, queue=awx_private_queue)
[2021-04-07 08:24:19,659: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6e1ae650-886d-4e8d-9d2f-d13565756ad5, queue=awx_private_queue)
2021-04-07 08:24:19,702 DEBUG awx.main.dispatch delivered 6e1ae650-886d-4e8d-9d2f-d13565756ad5 to worker[183] qsize 0
2021-04-07 08:24:19,722 DEBUG awx.main.dispatch task 6e1ae650-886d-4e8d-9d2f-d13565756ad5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:24:19,747 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:24:19,751 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:24:19,879 DEBUG awx.main.dispatch task 6e1ae650-886d-4e8d-9d2f-d13565756ad5 is finished
2021-04-07 08:24:28,687 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2b64ac8f-5d49-4f0c-a190-3ffb8287b5ae, queue=dev-awx-0)
[2021-04-07 08:24:28,687: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2b64ac8f-5d49-4f0c-a190-3ffb8287b5ae, queue=dev-awx-0)
2021-04-07 08:24:28,723 DEBUG awx.main.dispatch delivered 2b64ac8f-5d49-4f0c-a190-3ffb8287b5ae to worker[227] qsize 0
2021-04-07 08:24:28,737 DEBUG awx.main.dispatch task 2b64ac8f-5d49-4f0c-a190-3ffb8287b5ae starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:24:28,763 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139995040589376_2" does not exist
2021-04-07 08:24:28,771 DEBUG awx.main.dispatch task 2b64ac8f-5d49-4f0c-a190-3ffb8287b5ae is finished
2021-04-07 08:24:28,779 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cb13eabe-d7e4-4a0c-9700-752fe1eb37c1, queue=awx_private_queue)
[2021-04-07 08:24:28,779: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cb13eabe-d7e4-4a0c-9700-752fe1eb37c1, queue=awx_private_queue)
2021-04-07 08:24:28,810 DEBUG awx.main.dispatch delivered cb13eabe-d7e4-4a0c-9700-752fe1eb37c1 to worker[227] qsize 0
2021-04-07 08:24:28,837 DEBUG awx.main.dispatch task cb13eabe-d7e4-4a0c-9700-752fe1eb37c1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:24:28,849 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:24:28,851 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:24:28,857 DEBUG awx.main.dispatch task cb13eabe-d7e4-4a0c-9700-752fe1eb37c1 is finished
2021-04-07 08:24:48,792 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fcff6a0b-e26a-416a-ac75-5cad81656775, queue=awx_private_queue)
[2021-04-07 08:24:48,792: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fcff6a0b-e26a-416a-ac75-5cad81656775, queue=awx_private_queue)
2021-04-07 08:24:48,827 DEBUG awx.main.dispatch delivered fcff6a0b-e26a-416a-ac75-5cad81656775 to worker[198] qsize 0
2021-04-07 08:24:48,837 DEBUG awx.main.dispatch task fcff6a0b-e26a-416a-ac75-5cad81656775 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:24:48,875 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:24:48,878 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:24:48,884 DEBUG awx.main.dispatch task fcff6a0b-e26a-416a-ac75-5cad81656775 is finished
2021-04-07 08:24:49,622 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(46ead00e-7308-417b-b7cf-fe1588550cb0, queue=dev-awx-0)
[2021-04-07 08:24:49,622: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(46ead00e-7308-417b-b7cf-fe1588550cb0, queue=dev-awx-0)
2021-04-07 08:24:49,658 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e4153f51-5996-4f9a-860f-9669c791dfa3, queue=awx_private_queue)
[2021-04-07 08:24:49,658: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e4153f51-5996-4f9a-860f-9669c791dfa3, queue=awx_private_queue)
2021-04-07 08:24:49,663 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-07 08:24:49,692 DEBUG awx.main.dispatch delivered e4153f51-5996-4f9a-860f-9669c791dfa3 to worker[183] qsize 0
2021-04-07 08:24:49,700 DEBUG awx.main.dispatch task e4153f51-5996-4f9a-860f-9669c791dfa3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:24:49,714 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:24:49,716 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:24:49,720 DEBUG awx.main.dispatch task e4153f51-5996-4f9a-860f-9669c791dfa3 is finished
RESULT 2
OKREADY
2021-04-07 08:25:08,804 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6d17a9fb-b4d6-444e-aabf-f8b3165a68a6, queue=awx_private_queue)
[2021-04-07 08:25:08,804: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6d17a9fb-b4d6-444e-aabf-f8b3165a68a6, queue=awx_private_queue)
2021-04-07 08:25:08,846 DEBUG awx.main.dispatch delivered 6d17a9fb-b4d6-444e-aabf-f8b3165a68a6 to worker[224] qsize 0
2021-04-07 08:25:08,905 DEBUG awx.main.dispatch task 6d17a9fb-b4d6-444e-aabf-f8b3165a68a6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:25:08,937 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:25:08,939 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:25:08,949 DEBUG awx.main.dispatch task 6d17a9fb-b4d6-444e-aabf-f8b3165a68a6 is finished
2021-04-07 08:25:19,666 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(967f134e-10ac-41c2-bc55-7c362d05f02f, queue=awx_private_queue)
[2021-04-07 08:25:19,666: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(967f134e-10ac-41c2-bc55-7c362d05f02f, queue=awx_private_queue)
2021-04-07 08:25:19,717 DEBUG awx.main.dispatch delivered 967f134e-10ac-41c2-bc55-7c362d05f02f to worker[227] qsize 0
2021-04-07 08:25:19,728 DEBUG awx.main.dispatch task 967f134e-10ac-41c2-bc55-7c362d05f02f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:25:19,750 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:25:19,754 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:25:19,761 DEBUG awx.main.dispatch task 967f134e-10ac-41c2-bc55-7c362d05f02f is finished
2021-04-07 08:25:28,688 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(70cd78fc-68ca-4a96-bdfc-5f5073b33032, queue=dev-awx-0)
[2021-04-07 08:25:28,688: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(70cd78fc-68ca-4a96-bdfc-5f5073b33032, queue=dev-awx-0)
2021-04-07 08:25:28,743 DEBUG awx.main.dispatch delivered 70cd78fc-68ca-4a96-bdfc-5f5073b33032 to worker[224] qsize 0
2021-04-07 08:25:28,755 DEBUG awx.main.dispatch task 70cd78fc-68ca-4a96-bdfc-5f5073b33032 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:25:28,792 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139995040589376_3" does not exist
2021-04-07 08:25:28,798 DEBUG awx.main.dispatch task 70cd78fc-68ca-4a96-bdfc-5f5073b33032 is finished
2021-04-07 08:25:28,800 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fcaa9830-d957-4df1-a3d3-8fd468b59d73, queue=awx_private_queue)
[2021-04-07 08:25:28,800: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fcaa9830-d957-4df1-a3d3-8fd468b59d73, queue=awx_private_queue)
2021-04-07 08:25:28,828 DEBUG awx.main.dispatch delivered fcaa9830-d957-4df1-a3d3-8fd468b59d73 to worker[224] qsize 0
2021-04-07 08:25:28,836 DEBUG awx.main.dispatch task fcaa9830-d957-4df1-a3d3-8fd468b59d73 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:25:28,856 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:25:28,858 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:25:28,863 DEBUG awx.main.dispatch task fcaa9830-d957-4df1-a3d3-8fd468b59d73 is finished
2021-04-07 08:25:48,810 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(98e63aed-368e-4ded-9707-d8ffdba8c6a9, queue=awx_private_queue)
[2021-04-07 08:25:48,810: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(98e63aed-368e-4ded-9707-d8ffdba8c6a9, queue=awx_private_queue)
2021-04-07 08:25:48,843 DEBUG awx.main.dispatch delivered 98e63aed-368e-4ded-9707-d8ffdba8c6a9 to worker[227] qsize 0
2021-04-07 08:25:48,856 DEBUG awx.main.dispatch task 98e63aed-368e-4ded-9707-d8ffdba8c6a9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:25:48,882 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:25:48,885 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:25:48,891 DEBUG awx.main.dispatch task 98e63aed-368e-4ded-9707-d8ffdba8c6a9 is finished
2021-04-07 08:25:49,622 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(bc885433-8dec-4a91-8a86-9adc858afb89, queue=dev-awx-0)
[2021-04-07 08:25:49,622: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(bc885433-8dec-4a91-8a86-9adc858afb89, queue=dev-awx-0)
2021-04-07 08:25:49,683 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c84a40d6-f73f-47e5-8b82-94b2463ca066, queue=awx_private_queue)
[2021-04-07 08:25:49,683: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c84a40d6-f73f-47e5-8b82-94b2463ca066, queue=awx_private_queue)
2021-04-07 08:25:49,689 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-07 08:25:49,716 DEBUG awx.main.dispatch delivered c84a40d6-f73f-47e5-8b82-94b2463ca066 to worker[227] qsize 0
2021-04-07 08:25:49,723 DEBUG awx.main.dispatch task c84a40d6-f73f-47e5-8b82-94b2463ca066 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:25:49,734 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:25:49,736 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:25:49,741 DEBUG awx.main.dispatch task c84a40d6-f73f-47e5-8b82-94b2463ca066 is finished
2021-04-07 08:25:53,947 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(58a3eb88-73ee-4a7b-9b3f-b0cc81a0d524, queue=dev-awx-0)
[2021-04-07 08:25:53,947: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(58a3eb88-73ee-4a7b-9b3f-b0cc81a0d524, queue=dev-awx-0)
2021-04-07 08:25:53,976 DEBUG awx.main.dispatch delivered 58a3eb88-73ee-4a7b-9b3f-b0cc81a0d524 to worker[227] qsize 0
2021-04-07 08:25:53,982 DEBUG awx.main.dispatch task 58a3eb88-73ee-4a7b-9b3f-b0cc81a0d524 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-04-07 08:25:53,993 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-04-07 08:25:54,004 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 470, in awx_isolated_heartbeat
for isolated_instance in isolated_instance_qs:
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-04-07 08:25:54,008 DEBUG awx.main.dispatch task 58a3eb88-73ee-4a7b-9b3f-b0cc81a0d524 is finished
RESULT 2
OKREADY
2021-04-07 08:26:08,822 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cc0e296b-9518-4ac2-b5d0-f248e74cbebc, queue=awx_private_queue)
[2021-04-07 08:26:08,822: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cc0e296b-9518-4ac2-b5d0-f248e74cbebc, queue=awx_private_queue)
2021-04-07 08:26:08,864 DEBUG awx.main.dispatch delivered cc0e296b-9518-4ac2-b5d0-f248e74cbebc to worker[198] qsize 0
2021-04-07 08:26:08,883 DEBUG awx.main.dispatch task cc0e296b-9518-4ac2-b5d0-f248e74cbebc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:26:08,910 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:26:08,911 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:26:08,917 DEBUG awx.main.dispatch task cc0e296b-9518-4ac2-b5d0-f248e74cbebc is finished
2021-04-07 08:26:19,684 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e3f76939-f3a0-4543-9637-b781bea1f0f1, queue=awx_private_queue)
[2021-04-07 08:26:19,684: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e3f76939-f3a0-4543-9637-b781bea1f0f1, queue=awx_private_queue)
2021-04-07 08:26:19,713 DEBUG awx.main.dispatch delivered e3f76939-f3a0-4543-9637-b781bea1f0f1 to worker[224] qsize 0
2021-04-07 08:26:19,727 DEBUG awx.main.dispatch task e3f76939-f3a0-4543-9637-b781bea1f0f1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:26:19,755 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:26:19,759 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:26:19,769 DEBUG awx.main.dispatch task e3f76939-f3a0-4543-9637-b781bea1f0f1 is finished
2021-04-07 08:26:28,687 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(29f60993-a7b8-4dc9-989e-29bee293ce19, queue=dev-awx-0)
[2021-04-07 08:26:28,687: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(29f60993-a7b8-4dc9-989e-29bee293ce19, queue=dev-awx-0)
2021-04-07 08:26:28,724 DEBUG awx.main.dispatch delivered 29f60993-a7b8-4dc9-989e-29bee293ce19 to worker[198] qsize 0
2021-04-07 08:26:28,737 DEBUG awx.main.dispatch task 29f60993-a7b8-4dc9-989e-29bee293ce19 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:26:28,753 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139995040589376_2" does not exist
2021-04-07 08:26:28,757 DEBUG awx.main.dispatch task 29f60993-a7b8-4dc9-989e-29bee293ce19 is finished
2021-04-07 08:26:28,817 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6fdadd58-c2c3-441f-8b79-ccf0662e49e7, queue=awx_private_queue)
[2021-04-07 08:26:28,817: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6fdadd58-c2c3-441f-8b79-ccf0662e49e7, queue=awx_private_queue)
2021-04-07 08:26:28,848 DEBUG awx.main.dispatch delivered 6fdadd58-c2c3-441f-8b79-ccf0662e49e7 to worker[198] qsize 0
2021-04-07 08:26:28,854 DEBUG awx.main.dispatch task 6fdadd58-c2c3-441f-8b79-ccf0662e49e7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:26:28,874 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:26:28,876 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:26:28,882 DEBUG awx.main.dispatch task 6fdadd58-c2c3-441f-8b79-ccf0662e49e7 is finished
2021-04-07 08:26:48,820 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(81d21e2c-a1a4-4a35-b488-8c72fb356e35, queue=awx_private_queue)
[2021-04-07 08:26:48,820: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(81d21e2c-a1a4-4a35-b488-8c72fb356e35, queue=awx_private_queue)
2021-04-07 08:26:48,851 DEBUG awx.main.dispatch delivered 81d21e2c-a1a4-4a35-b488-8c72fb356e35 to worker[198] qsize 0
2021-04-07 08:26:48,866 DEBUG awx.main.dispatch task 81d21e2c-a1a4-4a35-b488-8c72fb356e35 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:26:48,891 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:26:48,893 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:26:48,901 DEBUG awx.main.dispatch task 81d21e2c-a1a4-4a35-b488-8c72fb356e35 is finished
2021-04-07 08:26:49,622 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(173cbbfe-3f76-43f3-a706-85d90a43bb03, queue=dev-awx-0)
[2021-04-07 08:26:49,622: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(173cbbfe-3f76-43f3-a706-85d90a43bb03, queue=dev-awx-0)
2021-04-07 08:26:49,659 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-07 08:26:49,683 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1147304b-fbf6-453b-935e-fd9551479876, queue=awx_private_queue)
[2021-04-07 08:26:49,683: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1147304b-fbf6-453b-935e-fd9551479876, queue=awx_private_queue)
2021-04-07 08:26:49,717 DEBUG awx.main.dispatch delivered 1147304b-fbf6-453b-935e-fd9551479876 to worker[227] qsize 0
2021-04-07 08:26:49,725 DEBUG awx.main.dispatch task 1147304b-fbf6-453b-935e-fd9551479876 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:26:49,748 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:26:49,751 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:26:49,755 DEBUG awx.main.dispatch task 1147304b-fbf6-453b-935e-fd9551479876 is finished
RESULT 2
OKREADY
2021-04-07 08:27:08,834 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2b21a7bb-494f-4c70-979b-16e97dc2887c, queue=awx_private_queue)
[2021-04-07 08:27:08,834: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2b21a7bb-494f-4c70-979b-16e97dc2887c, queue=awx_private_queue)
2021-04-07 08:27:08,894 DEBUG awx.main.dispatch delivered 2b21a7bb-494f-4c70-979b-16e97dc2887c to worker[227] qsize 0
2021-04-07 08:27:08,906 DEBUG awx.main.dispatch task 2b21a7bb-494f-4c70-979b-16e97dc2887c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:27:08,924 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:27:08,926 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:27:08,932 DEBUG awx.main.dispatch task 2b21a7bb-494f-4c70-979b-16e97dc2887c is finished
2021-04-07 08:27:19,685 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8d130f6a-63fa-4ac0-93c3-0c777b685e8a, queue=awx_private_queue)
[2021-04-07 08:27:19,685: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8d130f6a-63fa-4ac0-93c3-0c777b685e8a, queue=awx_private_queue)
2021-04-07 08:27:19,727 DEBUG awx.main.dispatch delivered 8d130f6a-63fa-4ac0-93c3-0c777b685e8a to worker[198] qsize 0
2021-04-07 08:27:19,742 DEBUG awx.main.dispatch task 8d130f6a-63fa-4ac0-93c3-0c777b685e8a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:27:19,759 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:27:19,762 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:27:19,766 DEBUG awx.main.dispatch task 8d130f6a-63fa-4ac0-93c3-0c777b685e8a is finished
2021-04-07 08:27:28,686 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d6714dec-841a-48f7-9f84-b7412ac9a5fb, queue=dev-awx-0)
[2021-04-07 08:27:28,686: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d6714dec-841a-48f7-9f84-b7412ac9a5fb, queue=dev-awx-0)
2021-04-07 08:27:28,711 DEBUG awx.main.dispatch delivered d6714dec-841a-48f7-9f84-b7412ac9a5fb to worker[227] qsize 0
2021-04-07 08:27:28,720 DEBUG awx.main.dispatch task d6714dec-841a-48f7-9f84-b7412ac9a5fb starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:27:28,741 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139995040589376_3" does not exist
2021-04-07 08:27:28,746 DEBUG awx.main.dispatch task d6714dec-841a-48f7-9f84-b7412ac9a5fb is finished
2021-04-07 08:27:28,827 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b396bd71-2dd3-47ca-9da7-7a49f4cb4c46, queue=awx_private_queue)
[2021-04-07 08:27:28,827: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b396bd71-2dd3-47ca-9da7-7a49f4cb4c46, queue=awx_private_queue)
2021-04-07 08:27:28,858 DEBUG awx.main.dispatch delivered b396bd71-2dd3-47ca-9da7-7a49f4cb4c46 to worker[224] qsize 0
2021-04-07 08:27:28,868 DEBUG awx.main.dispatch task b396bd71-2dd3-47ca-9da7-7a49f4cb4c46 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:27:28,902 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:27:28,904 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:27:28,910 DEBUG awx.main.dispatch task b396bd71-2dd3-47ca-9da7-7a49f4cb4c46 is finished
2021-04-07 08:27:48,838 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a04a1955-5a20-45db-ba5a-3add69911cb1, queue=awx_private_queue)
[2021-04-07 08:27:48,838: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a04a1955-5a20-45db-ba5a-3add69911cb1, queue=awx_private_queue)
2021-04-07 08:27:48,864 DEBUG awx.main.dispatch delivered a04a1955-5a20-45db-ba5a-3add69911cb1 to worker[198] qsize 0
2021-04-07 08:27:48,876 DEBUG awx.main.dispatch task a04a1955-5a20-45db-ba5a-3add69911cb1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:27:48,891 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:27:48,892 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:27:48,897 DEBUG awx.main.dispatch task a04a1955-5a20-45db-ba5a-3add69911cb1 is finished
2021-04-07 08:27:49,621 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(98c081bd-ec05-4afd-b6d1-98d491d11a77, queue=dev-awx-0)
[2021-04-07 08:27:49,621: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(98c081bd-ec05-4afd-b6d1-98d491d11a77, queue=dev-awx-0)
2021-04-07 08:27:49,653 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-07 08:27:49,683 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d98e172a-cc9e-4bc0-93ef-625006d2b366, queue=awx_private_queue)
[2021-04-07 08:27:49,683: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d98e172a-cc9e-4bc0-93ef-625006d2b366, queue=awx_private_queue)
2021-04-07 08:27:49,709 DEBUG awx.main.dispatch delivered d98e172a-cc9e-4bc0-93ef-625006d2b366 to worker[227] qsize 0
2021-04-07 08:27:49,718 DEBUG awx.main.dispatch task d98e172a-cc9e-4bc0-93ef-625006d2b366 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:27:49,736 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:27:49,739 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:27:49,749 DEBUG awx.main.dispatch task d98e172a-cc9e-4bc0-93ef-625006d2b366 is finished
RESULT 2
OKREADY
2021-04-07 08:28:08,852 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(86ae747b-a7d3-415b-93d2-02e8bc6a1993, queue=awx_private_queue)
[2021-04-07 08:28:08,852: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(86ae747b-a7d3-415b-93d2-02e8bc6a1993, queue=awx_private_queue)
2021-04-07 08:28:08,888 DEBUG awx.main.dispatch delivered 86ae747b-a7d3-415b-93d2-02e8bc6a1993 to worker[198] qsize 0
2021-04-07 08:28:08,899 DEBUG awx.main.dispatch task 86ae747b-a7d3-415b-93d2-02e8bc6a1993 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:28:08,913 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:28:08,915 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:28:08,922 DEBUG awx.main.dispatch task 86ae747b-a7d3-415b-93d2-02e8bc6a1993 is finished
2021-04-07 08:28:19,687 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9e881427-caa0-4f33-88cf-7a0a10774471, queue=awx_private_queue)
[2021-04-07 08:28:19,687: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9e881427-caa0-4f33-88cf-7a0a10774471, queue=awx_private_queue)
2021-04-07 08:28:19,723 DEBUG awx.main.dispatch delivered 9e881427-caa0-4f33-88cf-7a0a10774471 to worker[183] qsize 0
2021-04-07 08:28:19,741 DEBUG awx.main.dispatch task 9e881427-caa0-4f33-88cf-7a0a10774471 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:28:19,763 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:28:19,766 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:28:19,770 DEBUG awx.main.dispatch task 9e881427-caa0-4f33-88cf-7a0a10774471 is finished
2021-04-07 08:28:28,688 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(41ecf073-0582-4875-a682-410025e97e00, queue=dev-awx-0)
[2021-04-07 08:28:28,688: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(41ecf073-0582-4875-a682-410025e97e00, queue=dev-awx-0)
2021-04-07 08:28:28,721 DEBUG awx.main.dispatch delivered 41ecf073-0582-4875-a682-410025e97e00 to worker[224] qsize 0
2021-04-07 08:28:28,732 DEBUG awx.main.dispatch task 41ecf073-0582-4875-a682-410025e97e00 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:28:28,749 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139995040589376_4" does not exist
2021-04-07 08:28:28,754 DEBUG awx.main.dispatch task 41ecf073-0582-4875-a682-410025e97e00 is finished
2021-04-07 08:28:28,847 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6bb3353d-e080-4d1f-af64-9f53165fa332, queue=awx_private_queue)
[2021-04-07 08:28:28,847: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6bb3353d-e080-4d1f-af64-9f53165fa332, queue=awx_private_queue)
2021-04-07 08:28:28,882 DEBUG awx.main.dispatch delivered 6bb3353d-e080-4d1f-af64-9f53165fa332 to worker[183] qsize 0
2021-04-07 08:28:28,890 DEBUG awx.main.dispatch task 6bb3353d-e080-4d1f-af64-9f53165fa332 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:28:28,900 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:28:28,901 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:28:28,907 DEBUG awx.main.dispatch task 6bb3353d-e080-4d1f-af64-9f53165fa332 is finished
2021-04-07 08:28:48,854 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b0bd7c27-4f86-4a03-a3a4-624faaf60177, queue=awx_private_queue)
[2021-04-07 08:28:48,854: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b0bd7c27-4f86-4a03-a3a4-624faaf60177, queue=awx_private_queue)
2021-04-07 08:28:48,888 DEBUG awx.main.dispatch delivered b0bd7c27-4f86-4a03-a3a4-624faaf60177 to worker[227] qsize 0
2021-04-07 08:28:48,899 DEBUG awx.main.dispatch task b0bd7c27-4f86-4a03-a3a4-624faaf60177 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:28:48,922 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:28:48,925 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:28:48,931 DEBUG awx.main.dispatch task b0bd7c27-4f86-4a03-a3a4-624faaf60177 is finished
2021-04-07 08:28:49,622 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e46bad3a-b6f2-4cfc-b9e7-d8ed62a0e6c4, queue=dev-awx-0)
[2021-04-07 08:28:49,622: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e46bad3a-b6f2-4cfc-b9e7-d8ed62a0e6c4, queue=dev-awx-0)
2021-04-07 08:28:49,659 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-07 08:28:49,684 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(89f08761-436a-4633-b337-be9db166e5d6, queue=awx_private_queue)
[2021-04-07 08:28:49,684: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(89f08761-436a-4633-b337-be9db166e5d6, queue=awx_private_queue)
2021-04-07 08:28:49,713 DEBUG awx.main.dispatch delivered 89f08761-436a-4633-b337-be9db166e5d6 to worker[227] qsize 0
2021-04-07 08:28:49,739 DEBUG awx.main.dispatch task 89f08761-436a-4633-b337-be9db166e5d6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:28:49,753 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:28:49,764 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:28:49,772 DEBUG awx.main.dispatch task 89f08761-436a-4633-b337-be9db166e5d6 is finished
RESULT 2
OKREADY
2021-04-07 08:29:08,865 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6e992cfd-3736-4a86-b6a5-7afaea398eae, queue=awx_private_queue)
[2021-04-07 08:29:08,865: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6e992cfd-3736-4a86-b6a5-7afaea398eae, queue=awx_private_queue)
2021-04-07 08:29:08,899 DEBUG awx.main.dispatch delivered 6e992cfd-3736-4a86-b6a5-7afaea398eae to worker[227] qsize 0
2021-04-07 08:29:08,908 DEBUG awx.main.dispatch task 6e992cfd-3736-4a86-b6a5-7afaea398eae starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:29:08,919 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:29:08,921 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:29:08,925 DEBUG awx.main.dispatch task 6e992cfd-3736-4a86-b6a5-7afaea398eae is finished
2021-04-07 08:29:19,689 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4b2d4161-6b1e-423f-8c53-6b8328873bba, queue=awx_private_queue)
[2021-04-07 08:29:19,689: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4b2d4161-6b1e-423f-8c53-6b8328873bba, queue=awx_private_queue)
2021-04-07 08:29:19,733 DEBUG awx.main.dispatch delivered 4b2d4161-6b1e-423f-8c53-6b8328873bba to worker[224] qsize 0
2021-04-07 08:29:19,751 DEBUG awx.main.dispatch task 4b2d4161-6b1e-423f-8c53-6b8328873bba starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:29:19,782 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:29:19,785 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:29:19,790 DEBUG awx.main.dispatch task 4b2d4161-6b1e-423f-8c53-6b8328873bba is finished
2021-04-07 08:29:28,689 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(53c6da1c-8554-4c19-8f19-e756d7342325, queue=dev-awx-0)
[2021-04-07 08:29:28,689: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(53c6da1c-8554-4c19-8f19-e756d7342325, queue=dev-awx-0)
2021-04-07 08:29:28,725 DEBUG awx.main.dispatch delivered 53c6da1c-8554-4c19-8f19-e756d7342325 to worker[227] qsize 0
2021-04-07 08:29:28,742 DEBUG awx.main.dispatch task 53c6da1c-8554-4c19-8f19-e756d7342325 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:29:28,776 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139995040589376_4" does not exist
2021-04-07 08:29:28,784 DEBUG awx.main.dispatch task 53c6da1c-8554-4c19-8f19-e756d7342325 is finished
2021-04-07 08:29:28,862 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(af857a72-a7f7-4362-bf02-85a847ceeb16, queue=awx_private_queue)
[2021-04-07 08:29:28,862: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(af857a72-a7f7-4362-bf02-85a847ceeb16, queue=awx_private_queue)
2021-04-07 08:29:28,890 DEBUG awx.main.dispatch delivered af857a72-a7f7-4362-bf02-85a847ceeb16 to worker[198] qsize 0
2021-04-07 08:29:28,899 DEBUG awx.main.dispatch task af857a72-a7f7-4362-bf02-85a847ceeb16 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:29:28,919 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:29:28,920 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:29:28,924 DEBUG awx.main.dispatch task af857a72-a7f7-4362-bf02-85a847ceeb16 is finished
2021-04-07 08:29:48,874 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c9defbe8-6e68-4222-a3f5-6ee32369a295, queue=awx_private_queue)
[2021-04-07 08:29:48,874: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c9defbe8-6e68-4222-a3f5-6ee32369a295, queue=awx_private_queue)
2021-04-07 08:29:48,916 DEBUG awx.main.dispatch delivered c9defbe8-6e68-4222-a3f5-6ee32369a295 to worker[198] qsize 0
2021-04-07 08:29:48,925 DEBUG awx.main.dispatch task c9defbe8-6e68-4222-a3f5-6ee32369a295 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:29:48,946 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:29:48,948 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:29:48,952 DEBUG awx.main.dispatch task c9defbe8-6e68-4222-a3f5-6ee32369a295 is finished
2021-04-07 08:29:49,622 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6c74d2e3-3f0f-4de3-a2f1-476c11674923, queue=dev-awx-0)
[2021-04-07 08:29:49,622: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6c74d2e3-3f0f-4de3-a2f1-476c11674923, queue=dev-awx-0)
2021-04-07 08:29:49,655 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-07 08:29:49,684 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(37e238be-3b06-437e-9ab7-456e3da253bf, queue=awx_private_queue)
[2021-04-07 08:29:49,684: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(37e238be-3b06-437e-9ab7-456e3da253bf, queue=awx_private_queue)
2021-04-07 08:29:49,718 DEBUG awx.main.dispatch delivered 37e238be-3b06-437e-9ab7-456e3da253bf to worker[224] qsize 0
2021-04-07 08:29:49,725 DEBUG awx.main.dispatch task 37e238be-3b06-437e-9ab7-456e3da253bf starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:29:49,744 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:29:49,747 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:29:49,752 DEBUG awx.main.dispatch task 37e238be-3b06-437e-9ab7-456e3da253bf is finished
RESULT 2
OKREADY
2021-04-07 08:30:08,884 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8b9daa56-8004-4eeb-ad73-96f352818fce, queue=awx_private_queue)
[2021-04-07 08:30:08,884: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8b9daa56-8004-4eeb-ad73-96f352818fce, queue=awx_private_queue)
2021-04-07 08:30:08,929 DEBUG awx.main.dispatch delivered 8b9daa56-8004-4eeb-ad73-96f352818fce to worker[227] qsize 0
2021-04-07 08:30:08,942 DEBUG awx.main.dispatch task 8b9daa56-8004-4eeb-ad73-96f352818fce starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:30:08,974 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:30:08,976 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:30:08,982 DEBUG awx.main.dispatch task 8b9daa56-8004-4eeb-ad73-96f352818fce is finished
2021-04-07 08:30:19,688 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6bada39e-2ee9-40d5-9343-16afe0614020, queue=awx_private_queue)
[2021-04-07 08:30:19,688: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6bada39e-2ee9-40d5-9343-16afe0614020, queue=awx_private_queue)
2021-04-07 08:30:19,724 DEBUG awx.main.dispatch delivered 6bada39e-2ee9-40d5-9343-16afe0614020 to worker[183] qsize 0
2021-04-07 08:30:19,737 DEBUG awx.main.dispatch task 6bada39e-2ee9-40d5-9343-16afe0614020 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:30:19,764 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:30:19,767 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:30:19,774 DEBUG awx.main.dispatch task 6bada39e-2ee9-40d5-9343-16afe0614020 is finished
2021-04-07 08:30:28,687 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3c80dfb7-b100-40b0-ad88-c32cb64f8c09, queue=dev-awx-0)
[2021-04-07 08:30:28,687: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3c80dfb7-b100-40b0-ad88-c32cb64f8c09, queue=dev-awx-0)
2021-04-07 08:30:28,712 DEBUG awx.main.dispatch delivered 3c80dfb7-b100-40b0-ad88-c32cb64f8c09 to worker[183] qsize 0
2021-04-07 08:30:28,725 DEBUG awx.main.dispatch task 3c80dfb7-b100-40b0-ad88-c32cb64f8c09 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:30:28,739 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139995040589376_5" does not exist
2021-04-07 08:30:28,743 DEBUG awx.main.dispatch task 3c80dfb7-b100-40b0-ad88-c32cb64f8c09 is finished
2021-04-07 08:30:28,878 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(11b49572-f2fb-4ce2-9a9a-6580f8cbcbe9, queue=awx_private_queue)
[2021-04-07 08:30:28,878: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(11b49572-f2fb-4ce2-9a9a-6580f8cbcbe9, queue=awx_private_queue)
2021-04-07 08:30:28,902 DEBUG awx.main.dispatch delivered 11b49572-f2fb-4ce2-9a9a-6580f8cbcbe9 to worker[198] qsize 0
2021-04-07 08:30:28,908 DEBUG awx.main.dispatch task 11b49572-f2fb-4ce2-9a9a-6580f8cbcbe9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:30:28,928 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:30:28,929 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:30:28,934 DEBUG awx.main.dispatch task 11b49572-f2fb-4ce2-9a9a-6580f8cbcbe9 is finished
2021-04-07 08:30:48,889 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(622d0e1a-0533-42a9-93e2-3582ca947f30, queue=awx_private_queue)
[2021-04-07 08:30:48,889: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(622d0e1a-0533-42a9-93e2-3582ca947f30, queue=awx_private_queue)
2021-04-07 08:30:48,928 DEBUG awx.main.dispatch delivered 622d0e1a-0533-42a9-93e2-3582ca947f30 to worker[227] qsize 0
2021-04-07 08:30:48,938 DEBUG awx.main.dispatch task 622d0e1a-0533-42a9-93e2-3582ca947f30 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:30:48,969 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:30:48,972 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:30:48,978 DEBUG awx.main.dispatch task 622d0e1a-0533-42a9-93e2-3582ca947f30 is finished
2021-04-07 08:30:49,622 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f0df4015-2bd6-4451-8bcb-226f874d05ac, queue=dev-awx-0)
[2021-04-07 08:30:49,622: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f0df4015-2bd6-4451-8bcb-226f874d05ac, queue=dev-awx-0)
2021-04-07 08:30:49,661 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-07 08:30:49,685 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bbe14a28-18d4-4a0e-95fe-7bf4fb2f8aaa, queue=awx_private_queue)
[2021-04-07 08:30:49,685: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bbe14a28-18d4-4a0e-95fe-7bf4fb2f8aaa, queue=awx_private_queue)
2021-04-07 08:30:49,713 DEBUG awx.main.dispatch delivered bbe14a28-18d4-4a0e-95fe-7bf4fb2f8aaa to worker[224] qsize 0
2021-04-07 08:30:49,721 DEBUG awx.main.dispatch task bbe14a28-18d4-4a0e-95fe-7bf4fb2f8aaa starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:30:49,740 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:30:49,742 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:30:49,746 DEBUG awx.main.dispatch task bbe14a28-18d4-4a0e-95fe-7bf4fb2f8aaa is finished
RESULT 2
OKREADY
2021-04-07 08:31:08,893 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(99e7f1b5-ad54-4209-aa59-9d7a305ceb66, queue=awx_private_queue)
[2021-04-07 08:31:08,893: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(99e7f1b5-ad54-4209-aa59-9d7a305ceb66, queue=awx_private_queue)
2021-04-07 08:31:08,924 DEBUG awx.main.dispatch delivered 99e7f1b5-ad54-4209-aa59-9d7a305ceb66 to worker[198] qsize 0
2021-04-07 08:31:08,935 DEBUG awx.main.dispatch task 99e7f1b5-ad54-4209-aa59-9d7a305ceb66 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:31:08,948 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:31:08,949 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:31:08,953 DEBUG awx.main.dispatch task 99e7f1b5-ad54-4209-aa59-9d7a305ceb66 is finished
2021-04-07 08:31:19,691 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2f34f4af-c8ec-4f02-924c-971ff77539b6, queue=awx_private_queue)
[2021-04-07 08:31:19,691: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2f34f4af-c8ec-4f02-924c-971ff77539b6, queue=awx_private_queue)
2021-04-07 08:31:19,730 DEBUG awx.main.dispatch delivered 2f34f4af-c8ec-4f02-924c-971ff77539b6 to worker[198] qsize 0
2021-04-07 08:31:19,743 DEBUG awx.main.dispatch task 2f34f4af-c8ec-4f02-924c-971ff77539b6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:31:19,758 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:31:19,761 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:31:19,767 DEBUG awx.main.dispatch task 2f34f4af-c8ec-4f02-924c-971ff77539b6 is finished
2021-04-07 08:31:28,693 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b0aa7403-0443-463b-8fff-3cdce1f36c29, queue=dev-awx-0)
[2021-04-07 08:31:28,693: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b0aa7403-0443-463b-8fff-3cdce1f36c29, queue=dev-awx-0)
2021-04-07 08:31:28,723 DEBUG awx.main.dispatch delivered b0aa7403-0443-463b-8fff-3cdce1f36c29 to worker[183] qsize 0
2021-04-07 08:31:28,739 DEBUG awx.main.dispatch task b0aa7403-0443-463b-8fff-3cdce1f36c29 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:31:28,759 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139995040589376_6" does not exist
2021-04-07 08:31:28,763 DEBUG awx.main.dispatch task b0aa7403-0443-463b-8fff-3cdce1f36c29 is finished
2021-04-07 08:31:28,892 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(60e4b638-735b-42ab-b8e1-3abf53108072, queue=awx_private_queue)
[2021-04-07 08:31:28,892: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(60e4b638-735b-42ab-b8e1-3abf53108072, queue=awx_private_queue)
2021-04-07 08:31:28,926 DEBUG awx.main.dispatch delivered 60e4b638-735b-42ab-b8e1-3abf53108072 to worker[183] qsize 0
2021-04-07 08:31:28,933 DEBUG awx.main.dispatch task 60e4b638-735b-42ab-b8e1-3abf53108072 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:31:28,950 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:31:28,952 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:31:28,959 DEBUG awx.main.dispatch task 60e4b638-735b-42ab-b8e1-3abf53108072 is finished
2021-04-07 08:31:48,893 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1b0643b1-b2c0-4412-aa2f-447aac4885cc, queue=awx_private_queue)
[2021-04-07 08:31:48,893: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1b0643b1-b2c0-4412-aa2f-447aac4885cc, queue=awx_private_queue)
2021-04-07 08:31:48,924 DEBUG awx.main.dispatch delivered 1b0643b1-b2c0-4412-aa2f-447aac4885cc to worker[183] qsize 0
2021-04-07 08:31:48,939 DEBUG awx.main.dispatch task 1b0643b1-b2c0-4412-aa2f-447aac4885cc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:31:48,952 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:31:48,953 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:31:48,957 DEBUG awx.main.dispatch task 1b0643b1-b2c0-4412-aa2f-447aac4885cc is finished
2021-04-07 08:31:49,622 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(89e0a661-4b5c-4a22-8a9a-da6e0a84590d, queue=dev-awx-0)
[2021-04-07 08:31:49,622: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(89e0a661-4b5c-4a22-8a9a-da6e0a84590d, queue=dev-awx-0)
2021-04-07 08:31:49,651 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-07 08:31:49,688 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(765a2a72-b7e8-4bd4-a234-f720f5161393, queue=awx_private_queue)
[2021-04-07 08:31:49,688: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(765a2a72-b7e8-4bd4-a234-f720f5161393, queue=awx_private_queue)
2021-04-07 08:31:49,718 DEBUG awx.main.dispatch delivered 765a2a72-b7e8-4bd4-a234-f720f5161393 to worker[183] qsize 0
2021-04-07 08:31:49,723 DEBUG awx.main.dispatch task 765a2a72-b7e8-4bd4-a234-f720f5161393 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:31:49,735 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:31:49,737 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:31:49,742 DEBUG awx.main.dispatch task 765a2a72-b7e8-4bd4-a234-f720f5161393 is finished
RESULT 2
OKREADY
2021-04-07 08:32:08,902 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a4e969d1-fc99-48ee-8476-326699d66ac3, queue=awx_private_queue)
[2021-04-07 08:32:08,902: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a4e969d1-fc99-48ee-8476-326699d66ac3, queue=awx_private_queue)
2021-04-07 08:32:08,940 DEBUG awx.main.dispatch delivered a4e969d1-fc99-48ee-8476-326699d66ac3 to worker[183] qsize 0
2021-04-07 08:32:08,946 DEBUG awx.main.dispatch task a4e969d1-fc99-48ee-8476-326699d66ac3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:32:08,960 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:32:08,962 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:32:08,968 DEBUG awx.main.dispatch task a4e969d1-fc99-48ee-8476-326699d66ac3 is finished
2021-04-07 08:32:19,690 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f9f1a86f-7cda-474a-a32b-d96886af0328, queue=awx_private_queue)
[2021-04-07 08:32:19,690: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f9f1a86f-7cda-474a-a32b-d96886af0328, queue=awx_private_queue)
2021-04-07 08:32:19,726 DEBUG awx.main.dispatch delivered f9f1a86f-7cda-474a-a32b-d96886af0328 to worker[224] qsize 0
2021-04-07 08:32:19,738 DEBUG awx.main.dispatch task f9f1a86f-7cda-474a-a32b-d96886af0328 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:32:19,763 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:32:19,765 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:32:19,770 DEBUG awx.main.dispatch task f9f1a86f-7cda-474a-a32b-d96886af0328 is finished
2021-04-07 08:32:28,689 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(74d7f926-f068-4b78-92ab-00fe6cc6b704, queue=dev-awx-0)
[2021-04-07 08:32:28,689: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(74d7f926-f068-4b78-92ab-00fe6cc6b704, queue=dev-awx-0)
2021-04-07 08:32:28,730 DEBUG awx.main.dispatch delivered 74d7f926-f068-4b78-92ab-00fe6cc6b704 to worker[227] qsize 0
2021-04-07 08:32:28,747 DEBUG awx.main.dispatch task 74d7f926-f068-4b78-92ab-00fe6cc6b704 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:32:28,784 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139995040589376_5" does not exist
2021-04-07 08:32:28,788 DEBUG awx.main.dispatch task 74d7f926-f068-4b78-92ab-00fe6cc6b704 is finished
2021-04-07 08:32:28,901 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(de61c99d-62b0-4511-98aa-e4f0d800cbb2, queue=awx_private_queue)
[2021-04-07 08:32:28,901: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(de61c99d-62b0-4511-98aa-e4f0d800cbb2, queue=awx_private_queue)
2021-04-07 08:32:28,925 DEBUG awx.main.dispatch delivered de61c99d-62b0-4511-98aa-e4f0d800cbb2 to worker[183] qsize 0
2021-04-07 08:32:28,931 DEBUG awx.main.dispatch task de61c99d-62b0-4511-98aa-e4f0d800cbb2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:32:28,958 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:32:28,959 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:32:28,963 DEBUG awx.main.dispatch task de61c99d-62b0-4511-98aa-e4f0d800cbb2 is finished
2021-04-07 08:32:48,912 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b3f37159-2b22-44cf-ada6-5ed0d713c176, queue=awx_private_queue)
[2021-04-07 08:32:48,912: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b3f37159-2b22-44cf-ada6-5ed0d713c176, queue=awx_private_queue)
2021-04-07 08:32:48,963 DEBUG awx.main.dispatch delivered b3f37159-2b22-44cf-ada6-5ed0d713c176 to worker[183] qsize 0
2021-04-07 08:32:48,979 DEBUG awx.main.dispatch task b3f37159-2b22-44cf-ada6-5ed0d713c176 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:32:48,992 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:32:48,994 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:32:48,999 DEBUG awx.main.dispatch task b3f37159-2b22-44cf-ada6-5ed0d713c176 is finished
2021-04-07 08:32:49,622 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(ce94a7cc-710a-41ab-8532-1d68e161bbbe, queue=dev-awx-0)
[2021-04-07 08:32:49,622: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(ce94a7cc-710a-41ab-8532-1d68e161bbbe, queue=dev-awx-0)
2021-04-07 08:32:49,653 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-07 08:32:49,687 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b7676e78-14b8-4336-8298-66670c9a4da8, queue=awx_private_queue)
[2021-04-07 08:32:49,687: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b7676e78-14b8-4336-8298-66670c9a4da8, queue=awx_private_queue)
2021-04-07 08:32:49,712 DEBUG awx.main.dispatch delivered b7676e78-14b8-4336-8298-66670c9a4da8 to worker[227] qsize 0
2021-04-07 08:32:49,721 DEBUG awx.main.dispatch task b7676e78-14b8-4336-8298-66670c9a4da8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:32:49,744 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:32:49,748 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:32:49,756 DEBUG awx.main.dispatch task b7676e78-14b8-4336-8298-66670c9a4da8 is finished
RESULT 2
OKREADY
2021-04-07 08:33:08,921 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e200be05-57a6-4ca0-bbde-752c95c15264, queue=awx_private_queue)
[2021-04-07 08:33:08,921: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e200be05-57a6-4ca0-bbde-752c95c15264, queue=awx_private_queue)
2021-04-07 08:33:08,956 DEBUG awx.main.dispatch delivered e200be05-57a6-4ca0-bbde-752c95c15264 to worker[183] qsize 0
2021-04-07 08:33:08,969 DEBUG awx.main.dispatch task e200be05-57a6-4ca0-bbde-752c95c15264 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:33:08,991 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:33:08,993 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:33:08,998 DEBUG awx.main.dispatch task e200be05-57a6-4ca0-bbde-752c95c15264 is finished
2021-04-07 08:33:19,692 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b2731401-4171-47d0-9027-6c37201fa0d6, queue=awx_private_queue)
[2021-04-07 08:33:19,692: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b2731401-4171-47d0-9027-6c37201fa0d6, queue=awx_private_queue)
2021-04-07 08:33:19,744 DEBUG awx.main.dispatch delivered b2731401-4171-47d0-9027-6c37201fa0d6 to worker[183] qsize 0
2021-04-07 08:33:19,758 DEBUG awx.main.dispatch task b2731401-4171-47d0-9027-6c37201fa0d6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:33:19,776 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:33:19,780 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:33:19,787 DEBUG awx.main.dispatch task b2731401-4171-47d0-9027-6c37201fa0d6 is finished
2021-04-07 08:33:28,691 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5b42d2aa-7583-4a5a-be93-7ab618bca7e7, queue=dev-awx-0)
[2021-04-07 08:33:28,691: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5b42d2aa-7583-4a5a-be93-7ab618bca7e7, queue=dev-awx-0)
2021-04-07 08:33:28,729 DEBUG awx.main.dispatch delivered 5b42d2aa-7583-4a5a-be93-7ab618bca7e7 to worker[227] qsize 0
2021-04-07 08:33:28,739 DEBUG awx.main.dispatch task 5b42d2aa-7583-4a5a-be93-7ab618bca7e7 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-07 08:33:28,762 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139995040589376_6" does not exist
2021-04-07 08:33:28,770 DEBUG awx.main.dispatch task 5b42d2aa-7583-4a5a-be93-7ab618bca7e7 is finished
2021-04-07 08:33:28,919 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(417216f3-7899-4883-9c0e-fc5c28d9dfb0, queue=awx_private_queue)
[2021-04-07 08:33:28,919: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(417216f3-7899-4883-9c0e-fc5c28d9dfb0, queue=awx_private_queue)
2021-04-07 08:33:28,952 DEBUG awx.main.dispatch delivered 417216f3-7899-4883-9c0e-fc5c28d9dfb0 to worker[183] qsize 0
2021-04-07 08:33:28,960 DEBUG awx.main.dispatch task 417216f3-7899-4883-9c0e-fc5c28d9dfb0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:33:28,976 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:33:28,977 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:33:28,983 DEBUG awx.main.dispatch task 417216f3-7899-4883-9c0e-fc5c28d9dfb0 is finished
2021-04-07 08:33:48,935 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5752a753-83f0-47af-a522-ee5500b51f47, queue=awx_private_queue)
[2021-04-07 08:33:48,935: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5752a753-83f0-47af-a522-ee5500b51f47, queue=awx_private_queue)
2021-04-07 08:33:48,989 DEBUG awx.main.dispatch delivered 5752a753-83f0-47af-a522-ee5500b51f47 to worker[183] qsize 0
2021-04-07 08:33:49,003 DEBUG awx.main.dispatch task 5752a753-83f0-47af-a522-ee5500b51f47 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:33:49,028 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:33:49,030 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-07 08:33:49,035 DEBUG awx.main.dispatch task 5752a753-83f0-47af-a522-ee5500b51f47 is finished
2021-04-07 08:33:49,623 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b53c131f-3ea3-420b-8051-1238d3d825f0, queue=dev-awx-0)
[2021-04-07 08:33:49,623: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b53c131f-3ea3-420b-8051-1238d3d825f0, queue=dev-awx-0)
2021-04-07 08:33:49,671 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-07 08:33:49,688 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f54e0a55-4248-4a89-a48e-84719058a92a, queue=awx_private_queue)
[2021-04-07 08:33:49,688: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f54e0a55-4248-4a89-a48e-84719058a92a, queue=awx_private_queue)
2021-04-07 08:33:49,714 DEBUG awx.main.dispatch delivered f54e0a55-4248-4a89-a48e-84719058a92a to worker[224] qsize 0
2021-04-07 08:33:49,725 DEBUG awx.main.dispatch task f54e0a55-4248-4a89-a48e-84719058a92a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-07 08:33:49,747 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-07 08:33:49,749 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-07 08:33:49,753 DEBUG awx.main.dispatch task f54e0a55-4248-4a89-a48e-84719058a92a is finished
RESULT 2
OKREADY
2021-04-07 08:34:08,940 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d67cce41-18b2-4bcc-9189-f9ffad46218a, queue=awx_private_queue)
[2021-04-07 08:34:08,940: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d67cce41-18b2-4bcc-9189-f9ffad46218a, queue=awx_private_queue)
2021-04-07 08:34:08,969 DEBUG awx.main.dispatch delivered d67cce41-18b2-4bcc-9189-f9ffad46218a to worker[227] qsize 0
2021-04-07 08:34:08,979 DEBUG awx.main.dispatch task d67cce41-18b2-4bcc-9189-f9ffad46218a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-07 08:34:08,993 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-07 08:34:08,995 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=F