By type
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"elapsed": 0,
"match_groupdict": {},
"match_groups": [],
"path": null,
"port": 5432,
"search_regex": null,
"state": "started"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"db": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
}
Default organization added.
Demo Credential, Inventory, and Job Template added.
(changed: True)
2021-04-24 10:44:33,868 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(282df34b-0eb6-4218-9a7e-090bc975e7f0, queue=awx_private_queue)
Successfully registered instance dev-awx-0
(changed: True)
2021-04-24 10:44:37,273 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(ab9c91cb-ca82-47eb-b045-390a3157c039, queue=awx_private_queue)
2021-04-24 10:44:37,303 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(23445264-662e-4b94-bfd5-e30413710065, queue=awx_private_queue)
Creating instance group tower
(changed: True)
/usr/bin/launch_awx_task.sh: line 27: unset: `#!/bin/sh': not a valid identifier
2021-04-24 10:44:37,992 INFO RPC interface 'supervisor' initialized
2021-04-24 10:44:37,993 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2021-04-24 10:44:37,993 INFO supervisord started with pid 150
2021-04-24 10:44:38,997 INFO spawned: 'awx-config-watcher' with pid 153
2021-04-24 10:44:39,002 INFO spawned: 'dispatcher' with pid 154
2021-04-24 10:44:39,006 INFO spawned: 'callback-receiver' with pid 155
2021-04-24 10:44:39,013 INFO spawned: 'channels-worker' with pid 156
READY
2021-04-24 10:44:40,150 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-24 10:44:40,151 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-24 10:44:40,151 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-24 10:44:40,151 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-24 10:44:42,210 - INFO - runworker - Using single-threaded worker.
2021-04-24 10:44:42,210 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer)
2021-04-24 10:44:42,211 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive
2021-04-24 10:44:42,212 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-04-24 10:44:42,212 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-04-24 10:44:42,216 ERROR celery.beat Removing corrupted schedule file '/var/lib/awx/beat.db': error(11, 'Resource temporarily unavailable')
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 485, in setup_schedule
self._store = self._open_schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 475, in _open_schedule
return self.persistence.open(self.schedule_filename, writeback=True)
File "/usr/lib64/python3.6/shelve.py", line 243, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/usr/lib64/python3.6/shelve.py", line 227, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
File "/usr/lib64/python3.6/dbm/__init__.py", line 94, in open
return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable
2021-04-24 10:44:42,220 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2021-04-24 10:44:42,220 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2021-04-24 10:44:42,218 WARNING awx.main.dispatch scaling up worker pid:177
2021-04-24 10:44:42,228 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2021-04-24 10:44:42,228 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2021-04-24 10:44:42,236 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:181
2021-04-24 10:44:42,236 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:181
2021-04-24 10:44:42,233 WARNING awx.main.dispatch scaling up worker pid:180
2021-04-24 10:44:42,249 WARNING awx.main.dispatch scaling up worker pid:182
2021-04-24 10:44:42,263 WARNING awx.main.dispatch scaling up worker pid:183
2021-04-24 10:44:42,270 DEBUG awx.main.tasks Syncing Schedules
2021-04-24 10:44:42,459 DEBUG awx.main.tasks Waited 0.0009615421295166016 seconds to obtain lock name: cluster_policy_lock
2021-04-24 10:44:42,469 DEBUG awx.main.tasks Total non-isolated instances:1 available for policy: 1
2021-04-24 10:44:42,469 DEBUG awx.main.tasks Policy percentage, adding Instances [1] to Group tower
2021-04-24 10:44:42,469 DEBUG awx.main.tasks Adding instances [1] to group tower
2021-04-24 10:44:42,474 DEBUG awx.main.tasks Cluster policy computation finished in 0.01485896110534668 seconds
2021-04-24 10:44:42,475 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 10:44:42,503 DEBUG awx.main.dispatch listening on ['awx_private_queue [direct]', 'dev-awx-0 [direct]', 'tower_broadcast_all_dev-awx-0 [fanout]']
RESULT 2
OKREADY
2021-04-24 10:45:02,358 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(16453017-22ec-4ce3-89d5-60cd39ea859d, queue=awx_private_queue)
[2021-04-24 10:45:02,358: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(16453017-22ec-4ce3-89d5-60cd39ea859d, queue=awx_private_queue)
2021-04-24 10:45:02,376 DEBUG awx.main.dispatch delivered 16453017-22ec-4ce3-89d5-60cd39ea859d to worker[177] qsize 0
2021-04-24 10:45:02,381 DEBUG awx.main.dispatch task 16453017-22ec-4ce3-89d5-60cd39ea859d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:45:02,383 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:45:02,406 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:45:02,442 DEBUG awx.main.dispatch task 16453017-22ec-4ce3-89d5-60cd39ea859d is finished
2021-04-24 10:45:12,347 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(38a6b704-7181-4fe8-9d65-e86af83c140e, queue=awx_private_queue)
[2021-04-24 10:45:12,347: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(38a6b704-7181-4fe8-9d65-e86af83c140e, queue=awx_private_queue)
2021-04-24 10:45:12,378 DEBUG awx.main.dispatch delivered 38a6b704-7181-4fe8-9d65-e86af83c140e to worker[180] qsize 0
2021-04-24 10:45:12,385 DEBUG awx.main.dispatch task 38a6b704-7181-4fe8-9d65-e86af83c140e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:45:12,403 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:45:12,418 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:45:12.414819+00:00
2021-04-24 10:45:12,438 DEBUG awx.main.dispatch task 38a6b704-7181-4fe8-9d65-e86af83c140e is finished
2021-04-24 10:45:22,339 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(617d3a8b-bfba-4dd5-9ecc-7166cfb0a62a, queue=awx_private_queue)
[2021-04-24 10:45:22,339: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(617d3a8b-bfba-4dd5-9ecc-7166cfb0a62a, queue=awx_private_queue)
2021-04-24 10:45:22,357 DEBUG awx.main.dispatch delivered 617d3a8b-bfba-4dd5-9ecc-7166cfb0a62a to worker[183] qsize 0
2021-04-24 10:45:22,363 DEBUG awx.main.dispatch task 617d3a8b-bfba-4dd5-9ecc-7166cfb0a62a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:45:22,367 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:45:22,399 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:45:22,439 DEBUG awx.main.dispatch task 617d3a8b-bfba-4dd5-9ecc-7166cfb0a62a is finished
2021-04-24 10:45:42,349 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(37c3f5cf-2da0-4738-b108-bb7e256040cf, queue=dev-awx-0)
[2021-04-24 10:45:42,349: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(37c3f5cf-2da0-4738-b108-bb7e256040cf, queue=dev-awx-0)
2021-04-24 10:45:42,563 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ffccbca7-1ef7-4992-abd6-dd09421e7e23, queue=dev-awx-0)
[2021-04-24 10:45:42,563: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ffccbca7-1ef7-4992-abd6-dd09421e7e23, queue=dev-awx-0)
2021-04-24 10:45:42,577 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0314a10f-2450-490d-bf87-e7e07ce8f429, queue=awx_private_queue)
[2021-04-24 10:45:42,577: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0314a10f-2450-490d-bf87-e7e07ce8f429, queue=awx_private_queue)
2021-04-24 10:45:42,584 DEBUG awx.main.dispatch delivered 37c3f5cf-2da0-4738-b108-bb7e256040cf to worker[177] qsize 0
2021-04-24 10:45:42,588 DEBUG awx.main.dispatch delivered ffccbca7-1ef7-4992-abd6-dd09421e7e23 to worker[183] qsize 0
2021-04-24 10:45:42,591 DEBUG awx.main.dispatch delivered 0314a10f-2450-490d-bf87-e7e07ce8f429 to worker[182] qsize 0
2021-04-24 10:45:42,592 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(cb83651f-7996-41f1-9acd-6319c1193b26, queue=awx_private_queue)
[2021-04-24 10:45:42,592: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(cb83651f-7996-41f1-9acd-6319c1193b26, queue=awx_private_queue)
2021-04-24 10:45:42,590 DEBUG awx.main.dispatch task 37c3f5cf-2da0-4738-b108-bb7e256040cf starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 10:45:42,594 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 10:45:42,596 DEBUG awx.main.dispatch task 0314a10f-2450-490d-bf87-e7e07ce8f429 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:45:42,601 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:45:42,599 DEBUG awx.main.dispatch task ffccbca7-1ef7-4992-abd6-dd09421e7e23 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 10:45:42,603 DEBUG awx.main.dispatch delivered cb83651f-7996-41f1-9acd-6319c1193b26 to worker[180] qsize 0
2021-04-24 10:45:42,605 DEBUG awx.main.dispatch task cb83651f-7996-41f1-9acd-6319c1193b26 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:45:42,612 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:45:42,616 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:45:12.403359+00:00
2021-04-24 10:45:42,622 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:45:42,623 DEBUG awx.main.dispatch task ffccbca7-1ef7-4992-abd6-dd09421e7e23 is finished
2021-04-24 10:45:42,629 DEBUG awx.main.dispatch task cb83651f-7996-41f1-9acd-6319c1193b26 is finished
2021-04-24 10:45:42,637 DEBUG awx.main.dispatch task 37c3f5cf-2da0-4738-b108-bb7e256040cf is finished
2021-04-24 10:45:42,666 DEBUG awx.main.dispatch task 0314a10f-2450-490d-bf87-e7e07ce8f429 is finished
RESULT 2
OKREADY
2021-04-24 10:46:02,578 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4bb66caa-298f-4ea5-8176-ad7aeee918da, queue=awx_private_queue)
[2021-04-24 10:46:02,578: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4bb66caa-298f-4ea5-8176-ad7aeee918da, queue=awx_private_queue)
2021-04-24 10:46:02,597 DEBUG awx.main.dispatch delivered 4bb66caa-298f-4ea5-8176-ad7aeee918da to worker[183] qsize 0
2021-04-24 10:46:02,599 DEBUG awx.main.dispatch task 4bb66caa-298f-4ea5-8176-ad7aeee918da starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:46:02,600 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:46:02,616 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:46:02,644 DEBUG awx.main.dispatch task 4bb66caa-298f-4ea5-8176-ad7aeee918da is finished
2021-04-24 10:46:12,593 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(139937ce-c4e0-44a0-83b7-0c0ec81eb125, queue=awx_private_queue)
[2021-04-24 10:46:12,593: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(139937ce-c4e0-44a0-83b7-0c0ec81eb125, queue=awx_private_queue)
2021-04-24 10:46:12,605 DEBUG awx.main.dispatch delivered 139937ce-c4e0-44a0-83b7-0c0ec81eb125 to worker[177] qsize 0
2021-04-24 10:46:12,607 DEBUG awx.main.dispatch task 139937ce-c4e0-44a0-83b7-0c0ec81eb125 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:46:12,621 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:46:12,624 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:45:42.612974+00:00
2021-04-24 10:46:12,639 DEBUG awx.main.dispatch task 139937ce-c4e0-44a0-83b7-0c0ec81eb125 is finished
2021-04-24 10:46:22,582 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b8157509-5411-45ca-8cce-e6538b2913e6, queue=awx_private_queue)
[2021-04-24 10:46:22,582: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b8157509-5411-45ca-8cce-e6538b2913e6, queue=awx_private_queue)
2021-04-24 10:46:22,606 DEBUG awx.main.dispatch delivered b8157509-5411-45ca-8cce-e6538b2913e6 to worker[177] qsize 0
2021-04-24 10:46:22,612 DEBUG awx.main.dispatch task b8157509-5411-45ca-8cce-e6538b2913e6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:46:22,615 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:46:22,639 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:46:22,672 DEBUG awx.main.dispatch task b8157509-5411-45ca-8cce-e6538b2913e6 is finished
2021-04-24 10:46:42,359 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(060e830b-8812-4c75-ad01-71f04f2e289f, queue=dev-awx-0)
[2021-04-24 10:46:42,359: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(060e830b-8812-4c75-ad01-71f04f2e289f, queue=dev-awx-0)
2021-04-24 10:46:42,546 DEBUG awx.main.dispatch delivered 060e830b-8812-4c75-ad01-71f04f2e289f to worker[177] qsize 0
2021-04-24 10:46:42,547 DEBUG awx.main.dispatch task 060e830b-8812-4c75-ad01-71f04f2e289f starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 10:46:42,549 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 10:46:42,564 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5c0a9dd4-373f-4eb9-b064-fefa9c0d9a5a, queue=dev-awx-0)
[2021-04-24 10:46:42,564: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5c0a9dd4-373f-4eb9-b064-fefa9c0d9a5a, queue=dev-awx-0)
2021-04-24 10:46:42,568 DEBUG awx.main.dispatch task 060e830b-8812-4c75-ad01-71f04f2e289f is finished
2021-04-24 10:46:42,579 DEBUG awx.main.dispatch delivered 5c0a9dd4-373f-4eb9-b064-fefa9c0d9a5a to worker[182] qsize 0
2021-04-24 10:46:42,579 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5a0e261b-57a6-473a-8a6e-7a8f63e55d8a, queue=awx_private_queue)
[2021-04-24 10:46:42,579: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5a0e261b-57a6-473a-8a6e-7a8f63e55d8a, queue=awx_private_queue)
2021-04-24 10:46:42,582 DEBUG awx.main.dispatch task 5c0a9dd4-373f-4eb9-b064-fefa9c0d9a5a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 10:46:42,591 DEBUG awx.main.dispatch delivered 5a0e261b-57a6-473a-8a6e-7a8f63e55d8a to worker[180] qsize 0
2021-04-24 10:46:42,592 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a5c30811-4ed1-4575-8a5a-03b573e854f1, queue=awx_private_queue)
[2021-04-24 10:46:42,592: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a5c30811-4ed1-4575-8a5a-03b573e854f1, queue=awx_private_queue)
2021-04-24 10:46:42,597 DEBUG awx.main.dispatch task 5a0e261b-57a6-473a-8a6e-7a8f63e55d8a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:46:42,600 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:46:42,607 DEBUG awx.main.dispatch delivered a5c30811-4ed1-4575-8a5a-03b573e854f1 to worker[177] qsize 0
2021-04-24 10:46:42,609 DEBUG awx.main.dispatch task a5c30811-4ed1-4575-8a5a-03b573e854f1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:46:42,609 DEBUG awx.main.dispatch task 5c0a9dd4-373f-4eb9-b064-fefa9c0d9a5a is finished
2021-04-24 10:46:42,616 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:46:42,618 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:46:12.621289+00:00
2021-04-24 10:46:42,620 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:46:42,627 DEBUG awx.main.dispatch task a5c30811-4ed1-4575-8a5a-03b573e854f1 is finished
2021-04-24 10:46:42,661 DEBUG awx.main.dispatch task 5a0e261b-57a6-473a-8a6e-7a8f63e55d8a is finished
RESULT 2
OKREADY
2021-04-24 10:47:02,587 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3f09875f-0fb6-40f6-9769-f5088bc2db85, queue=awx_private_queue)
[2021-04-24 10:47:02,587: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3f09875f-0fb6-40f6-9769-f5088bc2db85, queue=awx_private_queue)
2021-04-24 10:47:02,599 DEBUG awx.main.dispatch delivered 3f09875f-0fb6-40f6-9769-f5088bc2db85 to worker[182] qsize 0
2021-04-24 10:47:02,601 DEBUG awx.main.dispatch task 3f09875f-0fb6-40f6-9769-f5088bc2db85 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:47:02,602 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:47:02,638 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:47:02,679 DEBUG awx.main.dispatch task 3f09875f-0fb6-40f6-9769-f5088bc2db85 is finished
2021-04-24 10:47:12,593 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(51c96640-6ba7-4ebb-8376-caa6b3b9fdd1, queue=awx_private_queue)
[2021-04-24 10:47:12,593: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(51c96640-6ba7-4ebb-8376-caa6b3b9fdd1, queue=awx_private_queue)
2021-04-24 10:47:12,608 DEBUG awx.main.dispatch delivered 51c96640-6ba7-4ebb-8376-caa6b3b9fdd1 to worker[180] qsize 0
2021-04-24 10:47:12,610 DEBUG awx.main.dispatch task 51c96640-6ba7-4ebb-8376-caa6b3b9fdd1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:47:12,637 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:47:12,642 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:46:42.616241+00:00
2021-04-24 10:47:12,689 DEBUG awx.main.dispatch task 51c96640-6ba7-4ebb-8376-caa6b3b9fdd1 is finished
2021-04-24 10:47:22,587 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(336376f2-f9d8-49aa-8a53-7feca2cc8760, queue=awx_private_queue)
[2021-04-24 10:47:22,587: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(336376f2-f9d8-49aa-8a53-7feca2cc8760, queue=awx_private_queue)
2021-04-24 10:47:22,602 DEBUG awx.main.dispatch delivered 336376f2-f9d8-49aa-8a53-7feca2cc8760 to worker[177] qsize 0
2021-04-24 10:47:22,603 DEBUG awx.main.dispatch task 336376f2-f9d8-49aa-8a53-7feca2cc8760 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:47:22,605 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:47:22,636 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:47:22,675 DEBUG awx.main.dispatch task 336376f2-f9d8-49aa-8a53-7feca2cc8760 is finished
2021-04-24 10:47:42,372 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(ff2acf10-3b23-4b7e-bfcd-26980944c88e, queue=dev-awx-0)
[2021-04-24 10:47:42,372: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(ff2acf10-3b23-4b7e-bfcd-26980944c88e, queue=dev-awx-0)
2021-04-24 10:47:42,564 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(80fa4408-0acf-4511-92f8-64a2ff0015c4, queue=dev-awx-0)
[2021-04-24 10:47:42,564: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(80fa4408-0acf-4511-92f8-64a2ff0015c4, queue=dev-awx-0)
2021-04-24 10:47:42,586 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0cb22797-fbe4-4344-a51e-9fd1041925cf, queue=awx_private_queue)
[2021-04-24 10:47:42,586: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0cb22797-fbe4-4344-a51e-9fd1041925cf, queue=awx_private_queue)
2021-04-24 10:47:42,600 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c609830e-87f6-46c3-80c8-adc7d34facf9, queue=awx_private_queue)
[2021-04-24 10:47:42,600: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c609830e-87f6-46c3-80c8-adc7d34facf9, queue=awx_private_queue)
2021-04-24 10:47:42,602 DEBUG awx.main.dispatch delivered ff2acf10-3b23-4b7e-bfcd-26980944c88e to worker[180] qsize 0
2021-04-24 10:47:42,606 DEBUG awx.main.dispatch delivered 80fa4408-0acf-4511-92f8-64a2ff0015c4 to worker[177] qsize 0
2021-04-24 10:47:42,608 DEBUG awx.main.dispatch delivered 0cb22797-fbe4-4344-a51e-9fd1041925cf to worker[183] qsize 0
2021-04-24 10:47:42,611 DEBUG awx.main.dispatch task 0cb22797-fbe4-4344-a51e-9fd1041925cf starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:47:42,617 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:47:42,612 DEBUG awx.main.dispatch task 80fa4408-0acf-4511-92f8-64a2ff0015c4 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 10:47:42,613 DEBUG awx.main.dispatch task ff2acf10-3b23-4b7e-bfcd-26980944c88e starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 10:47:42,624 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 10:47:42,633 DEBUG awx.main.dispatch delivered c609830e-87f6-46c3-80c8-adc7d34facf9 to worker[182] qsize 0
2021-04-24 10:47:42,645 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:47:42,641 DEBUG awx.main.dispatch task c609830e-87f6-46c3-80c8-adc7d34facf9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:47:42,656 DEBUG awx.main.dispatch task 80fa4408-0acf-4511-92f8-64a2ff0015c4 is finished
2021-04-24 10:47:42,673 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:47:42,674 DEBUG awx.main.dispatch task ff2acf10-3b23-4b7e-bfcd-26980944c88e is finished
2021-04-24 10:47:42,689 DEBUG awx.main.dispatch task 0cb22797-fbe4-4344-a51e-9fd1041925cf is finished
2021-04-24 10:47:42,686 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:47:12.637883+00:00
2021-04-24 10:47:42,721 DEBUG awx.main.dispatch task c609830e-87f6-46c3-80c8-adc7d34facf9 is finished
RESULT 2
OKREADY
2021-04-24 10:48:02,599 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d891f557-d32e-4138-aa95-456f5a3d6550, queue=awx_private_queue)
[2021-04-24 10:48:02,599: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d891f557-d32e-4138-aa95-456f5a3d6550, queue=awx_private_queue)
2021-04-24 10:48:02,615 DEBUG awx.main.dispatch delivered d891f557-d32e-4138-aa95-456f5a3d6550 to worker[180] qsize 0
2021-04-24 10:48:02,618 DEBUG awx.main.dispatch task d891f557-d32e-4138-aa95-456f5a3d6550 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:48:02,619 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:48:02,683 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:48:02,745 DEBUG awx.main.dispatch task d891f557-d32e-4138-aa95-456f5a3d6550 is finished
2021-04-24 10:48:12,601 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0a95b1cb-cb72-4515-be2c-966cdb72c63a, queue=awx_private_queue)
[2021-04-24 10:48:12,601: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0a95b1cb-cb72-4515-be2c-966cdb72c63a, queue=awx_private_queue)
2021-04-24 10:48:12,614 DEBUG awx.main.dispatch delivered 0a95b1cb-cb72-4515-be2c-966cdb72c63a to worker[180] qsize 0
2021-04-24 10:48:12,616 DEBUG awx.main.dispatch task 0a95b1cb-cb72-4515-be2c-966cdb72c63a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:48:12,629 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:48:12,632 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:47:42.673678+00:00
2021-04-24 10:48:12,665 DEBUG awx.main.dispatch task 0a95b1cb-cb72-4515-be2c-966cdb72c63a is finished
2021-04-24 10:48:22,599 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(04bd09cc-d9d3-475e-abe3-21c982ea9f56, queue=awx_private_queue)
[2021-04-24 10:48:22,599: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(04bd09cc-d9d3-475e-abe3-21c982ea9f56, queue=awx_private_queue)
2021-04-24 10:48:22,611 DEBUG awx.main.dispatch delivered 04bd09cc-d9d3-475e-abe3-21c982ea9f56 to worker[182] qsize 0
2021-04-24 10:48:22,613 DEBUG awx.main.dispatch task 04bd09cc-d9d3-475e-abe3-21c982ea9f56 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:48:22,614 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:48:22,633 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:48:22,673 DEBUG awx.main.dispatch task 04bd09cc-d9d3-475e-abe3-21c982ea9f56 is finished
2021-04-24 10:48:42,368 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(daec9035-2a0c-43a1-9b06-a8906e8b91f7, queue=dev-awx-0)
[2021-04-24 10:48:42,368: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(daec9035-2a0c-43a1-9b06-a8906e8b91f7, queue=dev-awx-0)
2021-04-24 10:48:42,420 DEBUG awx.main.dispatch delivered daec9035-2a0c-43a1-9b06-a8906e8b91f7 to worker[183] qsize 0
2021-04-24 10:48:42,422 DEBUG awx.main.dispatch task daec9035-2a0c-43a1-9b06-a8906e8b91f7 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 10:48:42,423 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 10:48:42,457 DEBUG awx.main.dispatch task daec9035-2a0c-43a1-9b06-a8906e8b91f7 is finished
2021-04-24 10:48:42,566 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(fb24bdd1-f431-49a5-b632-fc8475c68429, queue=dev-awx-0)
[2021-04-24 10:48:42,566: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(fb24bdd1-f431-49a5-b632-fc8475c68429, queue=dev-awx-0)
2021-04-24 10:48:42,741 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(310cacb9-90c4-4dd4-8882-3a611f28cfc7, queue=awx_private_queue)
[2021-04-24 10:48:42,741: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(310cacb9-90c4-4dd4-8882-3a611f28cfc7, queue=awx_private_queue)
2021-04-24 10:48:42,741 DEBUG awx.main.dispatch delivered fb24bdd1-f431-49a5-b632-fc8475c68429 to worker[183] qsize 0
2021-04-24 10:48:42,744 DEBUG awx.main.dispatch task fb24bdd1-f431-49a5-b632-fc8475c68429 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 10:48:42,758 DEBUG awx.main.dispatch delivered 310cacb9-90c4-4dd4-8882-3a611f28cfc7 to worker[180] qsize 0
2021-04-24 10:48:42,758 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(39fc48b8-6ad7-4d0d-893e-7a93dd903383, queue=awx_private_queue)
[2021-04-24 10:48:42,758: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(39fc48b8-6ad7-4d0d-893e-7a93dd903383, queue=awx_private_queue)
2021-04-24 10:48:42,760 DEBUG awx.main.dispatch task 310cacb9-90c4-4dd4-8882-3a611f28cfc7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:48:42,762 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:48:42,774 DEBUG awx.main.dispatch delivered 39fc48b8-6ad7-4d0d-893e-7a93dd903383 to worker[182] qsize 0
2021-04-24 10:48:42,775 DEBUG awx.main.dispatch task 39fc48b8-6ad7-4d0d-893e-7a93dd903383 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:48:42,778 DEBUG awx.main.dispatch task fb24bdd1-f431-49a5-b632-fc8475c68429 is finished
2021-04-24 10:48:42,789 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:48:42,791 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:48:12.629999+00:00
2021-04-24 10:48:42,792 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:48:42,809 DEBUG awx.main.dispatch task 39fc48b8-6ad7-4d0d-893e-7a93dd903383 is finished
2021-04-24 10:48:42,817 DEBUG awx.main.dispatch task 310cacb9-90c4-4dd4-8882-3a611f28cfc7 is finished
RESULT 2
OKREADY
2021-04-24 10:49:02,760 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2fa2441d-e967-4fe7-94fc-2a0e83df41fd, queue=awx_private_queue)
[2021-04-24 10:49:02,760: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2fa2441d-e967-4fe7-94fc-2a0e83df41fd, queue=awx_private_queue)
2021-04-24 10:49:02,789 DEBUG awx.main.dispatch delivered 2fa2441d-e967-4fe7-94fc-2a0e83df41fd to worker[183] qsize 0
2021-04-24 10:49:02,796 DEBUG awx.main.dispatch task 2fa2441d-e967-4fe7-94fc-2a0e83df41fd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:49:02,801 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:49:02,879 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:49:02,915 DEBUG awx.main.dispatch task 2fa2441d-e967-4fe7-94fc-2a0e83df41fd is finished
2021-04-24 10:49:12,764 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9f8a8e4e-eb3c-4717-90b9-a6c3c6ea0061, queue=awx_private_queue)
[2021-04-24 10:49:12,764: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9f8a8e4e-eb3c-4717-90b9-a6c3c6ea0061, queue=awx_private_queue)
2021-04-24 10:49:12,783 DEBUG awx.main.dispatch delivered 9f8a8e4e-eb3c-4717-90b9-a6c3c6ea0061 to worker[182] qsize 0
2021-04-24 10:49:12,787 DEBUG awx.main.dispatch task 9f8a8e4e-eb3c-4717-90b9-a6c3c6ea0061 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:49:12,843 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:49:12,850 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:48:42.789594+00:00
2021-04-24 10:49:12,888 DEBUG awx.main.dispatch task 9f8a8e4e-eb3c-4717-90b9-a6c3c6ea0061 is finished
2021-04-24 10:49:22,755 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4d78f3f4-ae0c-465e-a089-e293282d3b86, queue=awx_private_queue)
[2021-04-24 10:49:22,755: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4d78f3f4-ae0c-465e-a089-e293282d3b86, queue=awx_private_queue)
2021-04-24 10:49:22,774 DEBUG awx.main.dispatch delivered 4d78f3f4-ae0c-465e-a089-e293282d3b86 to worker[180] qsize 0
2021-04-24 10:49:22,778 DEBUG awx.main.dispatch task 4d78f3f4-ae0c-465e-a089-e293282d3b86 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:49:22,781 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:49:22,817 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:49:22,868 DEBUG awx.main.dispatch task 4d78f3f4-ae0c-465e-a089-e293282d3b86 is finished
2021-04-24 10:49:42,378 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(2f8d2def-8336-4f7a-ab17-90b7db360b4b, queue=dev-awx-0)
[2021-04-24 10:49:42,378: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(2f8d2def-8336-4f7a-ab17-90b7db360b4b, queue=dev-awx-0)
2021-04-24 10:49:42,423 DEBUG awx.main.dispatch delivered 2f8d2def-8336-4f7a-ab17-90b7db360b4b to worker[177] qsize 0
2021-04-24 10:49:42,566 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(125dd5b5-a1a9-4efb-b014-452483eabf0f, queue=dev-awx-0)
[2021-04-24 10:49:42,566: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(125dd5b5-a1a9-4efb-b014-452483eabf0f, queue=dev-awx-0)
2021-04-24 10:49:42,579 DEBUG awx.main.dispatch delivered 125dd5b5-a1a9-4efb-b014-452483eabf0f to worker[180] qsize 0
2021-04-24 10:49:42,581 DEBUG awx.main.dispatch task 125dd5b5-a1a9-4efb-b014-452483eabf0f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 10:49:42,426 DEBUG awx.main.dispatch task 2f8d2def-8336-4f7a-ab17-90b7db360b4b starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 10:49:42,604 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 10:49:42,604 DEBUG awx.main.dispatch task 125dd5b5-a1a9-4efb-b014-452483eabf0f is finished
2021-04-24 10:49:42,636 DEBUG awx.main.dispatch task 2f8d2def-8336-4f7a-ab17-90b7db360b4b is finished
2021-04-24 10:49:42,754 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(82090464-fc94-4f76-b1f8-ac5538d43283, queue=awx_private_queue)
[2021-04-24 10:49:42,754: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(82090464-fc94-4f76-b1f8-ac5538d43283, queue=awx_private_queue)
2021-04-24 10:49:42,779 DEBUG awx.main.dispatch delivered 82090464-fc94-4f76-b1f8-ac5538d43283 to worker[183] qsize 0
2021-04-24 10:49:42,780 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(30f8c0a7-5720-46b2-980d-22552de3aa24, queue=awx_private_queue)
[2021-04-24 10:49:42,780: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(30f8c0a7-5720-46b2-980d-22552de3aa24, queue=awx_private_queue)
2021-04-24 10:49:42,781 DEBUG awx.main.dispatch task 82090464-fc94-4f76-b1f8-ac5538d43283 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:49:42,783 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:49:42,796 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:49:42,801 DEBUG awx.main.dispatch delivered 30f8c0a7-5720-46b2-980d-22552de3aa24 to worker[177] qsize 0
2021-04-24 10:49:42,803 DEBUG awx.main.dispatch task 30f8c0a7-5720-46b2-980d-22552de3aa24 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:49:42,811 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:49:42,813 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:49:12.843594+00:00
2021-04-24 10:49:42,825 DEBUG awx.main.dispatch task 30f8c0a7-5720-46b2-980d-22552de3aa24 is finished
2021-04-24 10:49:42,828 DEBUG awx.main.dispatch task 82090464-fc94-4f76-b1f8-ac5538d43283 is finished
RESULT 2
OKREADY
2021-04-24 10:50:02,754 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3a337111-e6bf-4ca9-9687-f1b047a16b94, queue=awx_private_queue)
[2021-04-24 10:50:02,754: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3a337111-e6bf-4ca9-9687-f1b047a16b94, queue=awx_private_queue)
2021-04-24 10:50:02,769 DEBUG awx.main.dispatch delivered 3a337111-e6bf-4ca9-9687-f1b047a16b94 to worker[177] qsize 0
2021-04-24 10:50:02,771 DEBUG awx.main.dispatch task 3a337111-e6bf-4ca9-9687-f1b047a16b94 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:50:02,772 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:50:03,427 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:50:03,865 DEBUG awx.main.dispatch task 3a337111-e6bf-4ca9-9687-f1b047a16b94 is finished
2021-04-24 10:50:12,780 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(101e3556-9b43-487d-8e4d-ac9e42e612e3, queue=awx_private_queue)
[2021-04-24 10:50:12,780: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(101e3556-9b43-487d-8e4d-ac9e42e612e3, queue=awx_private_queue)
2021-04-24 10:50:12,794 DEBUG awx.main.dispatch delivered 101e3556-9b43-487d-8e4d-ac9e42e612e3 to worker[180] qsize 0
2021-04-24 10:50:12,795 DEBUG awx.main.dispatch task 101e3556-9b43-487d-8e4d-ac9e42e612e3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:50:12,812 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:50:12,814 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:49:42.811863+00:00
2021-04-24 10:50:12,828 DEBUG awx.main.dispatch task 101e3556-9b43-487d-8e4d-ac9e42e612e3 is finished
2021-04-24 10:50:22,754 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(de5d0101-1a1e-4c0d-bd63-b7ed5dd08965, queue=awx_private_queue)
[2021-04-24 10:50:22,754: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(de5d0101-1a1e-4c0d-bd63-b7ed5dd08965, queue=awx_private_queue)
2021-04-24 10:50:22,772 DEBUG awx.main.dispatch delivered de5d0101-1a1e-4c0d-bd63-b7ed5dd08965 to worker[182] qsize 0
2021-04-24 10:50:22,775 DEBUG awx.main.dispatch task de5d0101-1a1e-4c0d-bd63-b7ed5dd08965 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:50:22,778 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:50:22,796 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:50:22,831 DEBUG awx.main.dispatch task de5d0101-1a1e-4c0d-bd63-b7ed5dd08965 is finished
2021-04-24 10:50:42,392 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(44ff4fe7-907d-4d74-8083-d972dc25e5e4, queue=dev-awx-0)
[2021-04-24 10:50:42,392: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(44ff4fe7-907d-4d74-8083-d972dc25e5e4, queue=dev-awx-0)
2021-04-24 10:50:42,438 DEBUG awx.main.dispatch delivered 44ff4fe7-907d-4d74-8083-d972dc25e5e4 to worker[183] qsize 0
2021-04-24 10:50:42,449 DEBUG awx.main.dispatch task 44ff4fe7-907d-4d74-8083-d972dc25e5e4 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 10:50:42,454 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 10:50:42,481 DEBUG awx.main.dispatch task 44ff4fe7-907d-4d74-8083-d972dc25e5e4 is finished
2021-04-24 10:50:42,567 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(db0f961c-f578-4569-910c-976348eb18a0, queue=dev-awx-0)
[2021-04-24 10:50:42,567: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(db0f961c-f578-4569-910c-976348eb18a0, queue=dev-awx-0)
2021-04-24 10:50:42,584 DEBUG awx.main.dispatch delivered db0f961c-f578-4569-910c-976348eb18a0 to worker[180] qsize 0
2021-04-24 10:50:42,591 DEBUG awx.main.dispatch task db0f961c-f578-4569-910c-976348eb18a0 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 10:50:42,624 DEBUG awx.main.dispatch task db0f961c-f578-4569-910c-976348eb18a0 is finished
2021-04-24 10:50:42,755 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(edd34b9f-8e77-4aa5-96cf-31b1f0861cc2, queue=awx_private_queue)
[2021-04-24 10:50:42,755: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(edd34b9f-8e77-4aa5-96cf-31b1f0861cc2, queue=awx_private_queue)
2021-04-24 10:50:42,960 DEBUG awx.main.dispatch delivered edd34b9f-8e77-4aa5-96cf-31b1f0861cc2 to worker[182] qsize 0
2021-04-24 10:50:42,964 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(204c6272-d180-4fbc-834c-538565db1de7, queue=awx_private_queue)
[2021-04-24 10:50:42,964: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(204c6272-d180-4fbc-834c-538565db1de7, queue=awx_private_queue)
2021-04-24 10:50:42,964 DEBUG awx.main.dispatch task edd34b9f-8e77-4aa5-96cf-31b1f0861cc2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:50:42,967 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:50:42,983 DEBUG awx.main.dispatch delivered 204c6272-d180-4fbc-834c-538565db1de7 to worker[183] qsize 0
2021-04-24 10:50:42,985 DEBUG awx.main.dispatch task 204c6272-d180-4fbc-834c-538565db1de7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:50:42,990 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:50:42,993 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:50:42,995 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:50:12.812259+00:00
2021-04-24 10:50:43,013 DEBUG awx.main.dispatch task 204c6272-d180-4fbc-834c-538565db1de7 is finished
2021-04-24 10:50:43,015 DEBUG awx.main.dispatch task edd34b9f-8e77-4aa5-96cf-31b1f0861cc2 is finished
RESULT 2
OKREADY
2021-04-24 10:51:02,766 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(08e29d58-eedc-4127-81bb-3e05562520a3, queue=awx_private_queue)
[2021-04-24 10:51:02,766: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(08e29d58-eedc-4127-81bb-3e05562520a3, queue=awx_private_queue)
2021-04-24 10:51:02,782 DEBUG awx.main.dispatch delivered 08e29d58-eedc-4127-81bb-3e05562520a3 to worker[177] qsize 0
2021-04-24 10:51:02,786 DEBUG awx.main.dispatch task 08e29d58-eedc-4127-81bb-3e05562520a3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:51:02,791 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:51:02,814 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:51:02,847 DEBUG awx.main.dispatch task 08e29d58-eedc-4127-81bb-3e05562520a3 is finished
2021-04-24 10:51:12,965 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(03ee3083-b7a3-40b3-a3a0-423fa091a070, queue=awx_private_queue)
[2021-04-24 10:51:12,965: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(03ee3083-b7a3-40b3-a3a0-423fa091a070, queue=awx_private_queue)
2021-04-24 10:51:12,985 DEBUG awx.main.dispatch delivered 03ee3083-b7a3-40b3-a3a0-423fa091a070 to worker[182] qsize 0
2021-04-24 10:51:12,987 DEBUG awx.main.dispatch task 03ee3083-b7a3-40b3-a3a0-423fa091a070 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:51:12,999 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:51:13,003 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:50:42.993126+00:00
2021-04-24 10:51:13,023 DEBUG awx.main.dispatch task 03ee3083-b7a3-40b3-a3a0-423fa091a070 is finished
2021-04-24 10:51:22,767 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(db948283-8ba5-46ab-a026-592acdeb7db3, queue=awx_private_queue)
[2021-04-24 10:51:22,767: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(db948283-8ba5-46ab-a026-592acdeb7db3, queue=awx_private_queue)
2021-04-24 10:51:22,781 DEBUG awx.main.dispatch delivered db948283-8ba5-46ab-a026-592acdeb7db3 to worker[182] qsize 0
2021-04-24 10:51:22,783 DEBUG awx.main.dispatch task db948283-8ba5-46ab-a026-592acdeb7db3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:51:22,785 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:51:22,800 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:51:22,833 DEBUG awx.main.dispatch task db948283-8ba5-46ab-a026-592acdeb7db3 is finished
2021-04-24 10:51:42,394 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(fd6cf3f8-a4f0-4e2e-9509-cbfe5769aeee, queue=dev-awx-0)
[2021-04-24 10:51:42,394: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(fd6cf3f8-a4f0-4e2e-9509-cbfe5769aeee, queue=dev-awx-0)
2021-04-24 10:51:42,423 DEBUG awx.main.dispatch delivered fd6cf3f8-a4f0-4e2e-9509-cbfe5769aeee to worker[177] qsize 0
2021-04-24 10:51:42,425 DEBUG awx.main.dispatch task fd6cf3f8-a4f0-4e2e-9509-cbfe5769aeee starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 10:51:42,426 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 10:51:42,451 DEBUG awx.main.dispatch task fd6cf3f8-a4f0-4e2e-9509-cbfe5769aeee is finished
2021-04-24 10:51:42,567 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b94f861e-6d8d-4490-af38-c43aa86b0983, queue=dev-awx-0)
[2021-04-24 10:51:42,567: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b94f861e-6d8d-4490-af38-c43aa86b0983, queue=dev-awx-0)
2021-04-24 10:51:42,625 DEBUG awx.main.dispatch delivered b94f861e-6d8d-4490-af38-c43aa86b0983 to worker[177] qsize 0
2021-04-24 10:51:42,627 DEBUG awx.main.dispatch task b94f861e-6d8d-4490-af38-c43aa86b0983 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 10:51:42,640 DEBUG awx.main.dispatch task b94f861e-6d8d-4490-af38-c43aa86b0983 is finished
2021-04-24 10:51:42,767 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a4a85eec-314e-452f-b1fd-a2383c9b64b7, queue=awx_private_queue)
[2021-04-24 10:51:42,767: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a4a85eec-314e-452f-b1fd-a2383c9b64b7, queue=awx_private_queue)
2021-04-24 10:51:42,966 DEBUG awx.main.dispatch delivered a4a85eec-314e-452f-b1fd-a2383c9b64b7 to worker[177] qsize 0
2021-04-24 10:51:42,967 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(80e3323c-a1e0-4976-8841-66c564a6c53f, queue=awx_private_queue)
[2021-04-24 10:51:42,967: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(80e3323c-a1e0-4976-8841-66c564a6c53f, queue=awx_private_queue)
2021-04-24 10:51:42,969 DEBUG awx.main.dispatch task a4a85eec-314e-452f-b1fd-a2383c9b64b7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:51:42,970 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:51:42,984 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:51:42,984 DEBUG awx.main.dispatch delivered 80e3323c-a1e0-4976-8841-66c564a6c53f to worker[180] qsize 0
2021-04-24 10:51:42,986 DEBUG awx.main.dispatch task 80e3323c-a1e0-4976-8841-66c564a6c53f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:51:43,000 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:51:43,003 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:51:12.999548+00:00
2021-04-24 10:51:43,015 DEBUG awx.main.dispatch task 80e3323c-a1e0-4976-8841-66c564a6c53f is finished
2021-04-24 10:51:43,021 DEBUG awx.main.dispatch task a4a85eec-314e-452f-b1fd-a2383c9b64b7 is finished
RESULT 2
OKREADY
2021-04-24 10:52:02,769 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4ee288d4-2b9b-42bd-9f4b-7b3ff16abd24, queue=awx_private_queue)
[2021-04-24 10:52:02,769: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4ee288d4-2b9b-42bd-9f4b-7b3ff16abd24, queue=awx_private_queue)
2021-04-24 10:52:02,790 DEBUG awx.main.dispatch delivered 4ee288d4-2b9b-42bd-9f4b-7b3ff16abd24 to worker[180] qsize 0
2021-04-24 10:52:02,793 DEBUG awx.main.dispatch task 4ee288d4-2b9b-42bd-9f4b-7b3ff16abd24 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:52:02,794 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:52:02,811 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:52:02,845 DEBUG awx.main.dispatch task 4ee288d4-2b9b-42bd-9f4b-7b3ff16abd24 is finished
2021-04-24 10:52:12,972 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4434cd52-df43-45e9-b20c-48bfb1a3df80, queue=awx_private_queue)
[2021-04-24 10:52:12,972: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4434cd52-df43-45e9-b20c-48bfb1a3df80, queue=awx_private_queue)
2021-04-24 10:52:13,002 DEBUG awx.main.dispatch delivered 4434cd52-df43-45e9-b20c-48bfb1a3df80 to worker[183] qsize 0
2021-04-24 10:52:13,012 DEBUG awx.main.dispatch task 4434cd52-df43-45e9-b20c-48bfb1a3df80 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:52:13,027 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:52:13,033 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:51:43.001085+00:00
2021-04-24 10:52:13,057 DEBUG awx.main.dispatch task 4434cd52-df43-45e9-b20c-48bfb1a3df80 is finished
2021-04-24 10:52:22,767 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f02eb277-4416-46d9-9364-7a40449cfbbc, queue=awx_private_queue)
[2021-04-24 10:52:22,767: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f02eb277-4416-46d9-9364-7a40449cfbbc, queue=awx_private_queue)
2021-04-24 10:52:22,790 DEBUG awx.main.dispatch delivered f02eb277-4416-46d9-9364-7a40449cfbbc to worker[183] qsize 0
2021-04-24 10:52:22,792 DEBUG awx.main.dispatch task f02eb277-4416-46d9-9364-7a40449cfbbc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:52:22,793 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:52:22,812 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:52:22,839 DEBUG awx.main.dispatch task f02eb277-4416-46d9-9364-7a40449cfbbc is finished
2021-04-24 10:52:42,403 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(2d580a56-18da-42b8-a1c0-2ba7d70076c6, queue=dev-awx-0)
[2021-04-24 10:52:42,403: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(2d580a56-18da-42b8-a1c0-2ba7d70076c6, queue=dev-awx-0)
2021-04-24 10:52:42,436 DEBUG awx.main.dispatch delivered 2d580a56-18da-42b8-a1c0-2ba7d70076c6 to worker[180] qsize 0
2021-04-24 10:52:42,440 DEBUG awx.main.dispatch task 2d580a56-18da-42b8-a1c0-2ba7d70076c6 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 10:52:42,443 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 10:52:42,477 DEBUG awx.main.dispatch task 2d580a56-18da-42b8-a1c0-2ba7d70076c6 is finished
2021-04-24 10:52:42,567 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(63919bf4-7f61-4833-9bdb-909bf247c71d, queue=dev-awx-0)
[2021-04-24 10:52:42,567: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(63919bf4-7f61-4833-9bdb-909bf247c71d, queue=dev-awx-0)
2021-04-24 10:52:42,628 DEBUG awx.main.dispatch delivered 63919bf4-7f61-4833-9bdb-909bf247c71d to worker[182] qsize 0
2021-04-24 10:52:42,636 DEBUG awx.main.dispatch task 63919bf4-7f61-4833-9bdb-909bf247c71d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 10:52:42,856 DEBUG awx.main.dispatch task 63919bf4-7f61-4833-9bdb-909bf247c71d is finished
2021-04-24 10:52:42,767 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(203684e5-e0df-431c-b6c1-a610b30a91c3, queue=awx_private_queue)
[2021-04-24 10:52:42,767: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(203684e5-e0df-431c-b6c1-a610b30a91c3, queue=awx_private_queue)
2021-04-24 10:52:42,952 DEBUG awx.main.dispatch delivered 203684e5-e0df-431c-b6c1-a610b30a91c3 to worker[182] qsize 0
2021-04-24 10:52:42,954 DEBUG awx.main.dispatch task 203684e5-e0df-431c-b6c1-a610b30a91c3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:52:42,955 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:52:42,968 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3add4da2-7200-4bf1-af02-28ff43c181aa, queue=awx_private_queue)
[2021-04-24 10:52:42,968: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3add4da2-7200-4bf1-af02-28ff43c181aa, queue=awx_private_queue)
2021-04-24 10:52:42,972 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:52:42,989 DEBUG awx.main.dispatch delivered 3add4da2-7200-4bf1-af02-28ff43c181aa to worker[183] qsize 0
2021-04-24 10:52:42,994 DEBUG awx.main.dispatch task 3add4da2-7200-4bf1-af02-28ff43c181aa starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:52:43,002 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:52:43,005 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:52:13.027568+00:00
2021-04-24 10:52:43,011 DEBUG awx.main.dispatch task 203684e5-e0df-431c-b6c1-a610b30a91c3 is finished
2021-04-24 10:52:43,015 DEBUG awx.main.dispatch task 3add4da2-7200-4bf1-af02-28ff43c181aa is finished
RESULT 2
OKREADY
2021-04-24 10:53:02,771 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6efc9f24-5896-4bbc-ab92-f2b98f6c747c, queue=awx_private_queue)
[2021-04-24 10:53:02,771: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6efc9f24-5896-4bbc-ab92-f2b98f6c747c, queue=awx_private_queue)
2021-04-24 10:53:02,792 DEBUG awx.main.dispatch delivered 6efc9f24-5896-4bbc-ab92-f2b98f6c747c to worker[183] qsize 0
2021-04-24 10:53:02,794 DEBUG awx.main.dispatch task 6efc9f24-5896-4bbc-ab92-f2b98f6c747c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:53:02,796 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:53:02,823 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:53:02,872 DEBUG awx.main.dispatch task 6efc9f24-5896-4bbc-ab92-f2b98f6c747c is finished
2021-04-24 10:53:12,972 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(db3150b2-c045-4295-b2e3-4c2d54113770, queue=awx_private_queue)
[2021-04-24 10:53:12,972: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(db3150b2-c045-4295-b2e3-4c2d54113770, queue=awx_private_queue)
2021-04-24 10:53:12,999 DEBUG awx.main.dispatch delivered db3150b2-c045-4295-b2e3-4c2d54113770 to worker[180] qsize 0
2021-04-24 10:53:13,006 DEBUG awx.main.dispatch task db3150b2-c045-4295-b2e3-4c2d54113770 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:53:13,023 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:53:13,028 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:52:43.002428+00:00
2021-04-24 10:53:13,043 DEBUG awx.main.dispatch task db3150b2-c045-4295-b2e3-4c2d54113770 is finished
2021-04-24 10:53:22,771 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(20f42f42-a271-4af5-adec-12ace31fa167, queue=awx_private_queue)
[2021-04-24 10:53:22,771: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(20f42f42-a271-4af5-adec-12ace31fa167, queue=awx_private_queue)
2021-04-24 10:53:22,784 DEBUG awx.main.dispatch delivered 20f42f42-a271-4af5-adec-12ace31fa167 to worker[182] qsize 0
2021-04-24 10:53:22,792 DEBUG awx.main.dispatch task 20f42f42-a271-4af5-adec-12ace31fa167 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:53:22,801 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:53:22,832 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:53:22,872 DEBUG awx.main.dispatch task 20f42f42-a271-4af5-adec-12ace31fa167 is finished
2021-04-24 10:53:42,413 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(df9da330-2fe1-4093-9e72-11973c86f264, queue=dev-awx-0)
[2021-04-24 10:53:42,413: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(df9da330-2fe1-4093-9e72-11973c86f264, queue=dev-awx-0)
2021-04-24 10:53:42,444 DEBUG awx.main.dispatch delivered df9da330-2fe1-4093-9e72-11973c86f264 to worker[177] qsize 0
2021-04-24 10:53:42,447 DEBUG awx.main.dispatch task df9da330-2fe1-4093-9e72-11973c86f264 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 10:53:42,452 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 10:53:42,497 DEBUG awx.main.dispatch task df9da330-2fe1-4093-9e72-11973c86f264 is finished
2021-04-24 10:53:42,567 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(eda7563a-fa09-4a5a-8cf4-5b8d0596bd96, queue=dev-awx-0)
[2021-04-24 10:53:42,567: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(eda7563a-fa09-4a5a-8cf4-5b8d0596bd96, queue=dev-awx-0)
2021-04-24 10:53:42,580 DEBUG awx.main.dispatch delivered eda7563a-fa09-4a5a-8cf4-5b8d0596bd96 to worker[182] qsize 0
2021-04-24 10:53:42,582 DEBUG awx.main.dispatch task eda7563a-fa09-4a5a-8cf4-5b8d0596bd96 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 10:53:42,602 DEBUG awx.main.dispatch task eda7563a-fa09-4a5a-8cf4-5b8d0596bd96 is finished
2021-04-24 10:53:42,771 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fe03df4d-acc0-4fef-8b3f-b9348cbf250b, queue=awx_private_queue)
[2021-04-24 10:53:42,771: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fe03df4d-acc0-4fef-8b3f-b9348cbf250b, queue=awx_private_queue)
2021-04-24 10:53:42,989 DEBUG awx.main.dispatch delivered fe03df4d-acc0-4fef-8b3f-b9348cbf250b to worker[183] qsize 0
2021-04-24 10:53:42,990 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7078bb33-c543-46ac-8fad-9b63556260ed, queue=awx_private_queue)
[2021-04-24 10:53:42,990: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7078bb33-c543-46ac-8fad-9b63556260ed, queue=awx_private_queue)
2021-04-24 10:53:42,992 DEBUG awx.main.dispatch task fe03df4d-acc0-4fef-8b3f-b9348cbf250b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:53:42,995 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:53:43,004 DEBUG awx.main.dispatch delivered 7078bb33-c543-46ac-8fad-9b63556260ed to worker[177] qsize 0
2021-04-24 10:53:43,007 DEBUG awx.main.dispatch task 7078bb33-c543-46ac-8fad-9b63556260ed starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:53:43,014 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:53:43,015 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:53:43,016 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:53:13.023247+00:00
2021-04-24 10:53:43,028 DEBUG awx.main.dispatch task 7078bb33-c543-46ac-8fad-9b63556260ed is finished
2021-04-24 10:53:43,053 DEBUG awx.main.dispatch task fe03df4d-acc0-4fef-8b3f-b9348cbf250b is finished
RESULT 2
OKREADY
2021-04-24 10:54:02,787 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b0c4c5e4-b589-41d0-86d1-f862d6d531d7, queue=awx_private_queue)
[2021-04-24 10:54:02,787: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b0c4c5e4-b589-41d0-86d1-f862d6d531d7, queue=awx_private_queue)
2021-04-24 10:54:02,812 DEBUG awx.main.dispatch delivered b0c4c5e4-b589-41d0-86d1-f862d6d531d7 to worker[183] qsize 0
2021-04-24 10:54:02,819 DEBUG awx.main.dispatch task b0c4c5e4-b589-41d0-86d1-f862d6d531d7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:54:02,823 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:54:02,846 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:54:02,877 DEBUG awx.main.dispatch task b0c4c5e4-b589-41d0-86d1-f862d6d531d7 is finished
2021-04-24 10:54:12,990 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(638cdf2e-e5e7-491d-bfc3-4ea46a6c7626, queue=awx_private_queue)
[2021-04-24 10:54:12,990: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(638cdf2e-e5e7-491d-bfc3-4ea46a6c7626, queue=awx_private_queue)
2021-04-24 10:54:13,006 DEBUG awx.main.dispatch delivered 638cdf2e-e5e7-491d-bfc3-4ea46a6c7626 to worker[180] qsize 0
2021-04-24 10:54:13,009 DEBUG awx.main.dispatch task 638cdf2e-e5e7-491d-bfc3-4ea46a6c7626 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:54:13,021 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:54:13,026 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:53:43.014611+00:00
2021-04-24 10:54:13,037 DEBUG awx.main.dispatch task 638cdf2e-e5e7-491d-bfc3-4ea46a6c7626 is finished
2021-04-24 10:54:22,786 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0a8bbfa1-e821-4054-8ddb-16f54fa8d03c, queue=awx_private_queue)
[2021-04-24 10:54:22,786: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0a8bbfa1-e821-4054-8ddb-16f54fa8d03c, queue=awx_private_queue)
2021-04-24 10:54:22,806 DEBUG awx.main.dispatch delivered 0a8bbfa1-e821-4054-8ddb-16f54fa8d03c to worker[177] qsize 0
2021-04-24 10:54:22,809 DEBUG awx.main.dispatch task 0a8bbfa1-e821-4054-8ddb-16f54fa8d03c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:54:22,812 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:54:22,833 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:54:22,871 DEBUG awx.main.dispatch task 0a8bbfa1-e821-4054-8ddb-16f54fa8d03c is finished
2021-04-24 10:54:42,347 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(0836b12a-73a5-46f1-9dde-513c59bbdcb3, queue=dev-awx-0)
[2021-04-24 10:54:42,347: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(0836b12a-73a5-46f1-9dde-513c59bbdcb3, queue=dev-awx-0)
2021-04-24 10:54:42,363 DEBUG awx.main.dispatch delivered 0836b12a-73a5-46f1-9dde-513c59bbdcb3 to worker[183] qsize 0
2021-04-24 10:54:42,366 DEBUG awx.main.dispatch task 0836b12a-73a5-46f1-9dde-513c59bbdcb3 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-04-24 10:54:42,369 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-04-24 10:54:42,405 DEBUG awx.main.dispatch task 0836b12a-73a5-46f1-9dde-513c59bbdcb3 is finished
2021-04-24 10:54:42,413 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(bc93145d-422e-46cb-bc1d-42ec823d1726, queue=dev-awx-0)
[2021-04-24 10:54:42,413: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(bc93145d-422e-46cb-bc1d-42ec823d1726, queue=dev-awx-0)
2021-04-24 10:54:42,466 DEBUG awx.main.dispatch delivered bc93145d-422e-46cb-bc1d-42ec823d1726 to worker[182] qsize 0
2021-04-24 10:54:42,471 DEBUG awx.main.dispatch task bc93145d-422e-46cb-bc1d-42ec823d1726 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 10:54:42,479 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 10:54:42,518 DEBUG awx.main.dispatch task bc93145d-422e-46cb-bc1d-42ec823d1726 is finished
2021-04-24 10:54:42,567 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(19eb4926-78ad-4105-b420-b36491fefc94, queue=dev-awx-0)
[2021-04-24 10:54:42,567: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(19eb4926-78ad-4105-b420-b36491fefc94, queue=dev-awx-0)
2021-04-24 10:54:42,610 DEBUG awx.main.dispatch delivered 19eb4926-78ad-4105-b420-b36491fefc94 to worker[180] qsize 0
2021-04-24 10:54:42,612 DEBUG awx.main.dispatch task 19eb4926-78ad-4105-b420-b36491fefc94 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 10:54:42,624 DEBUG awx.main.dispatch task 19eb4926-78ad-4105-b420-b36491fefc94 is finished
2021-04-24 10:54:42,786 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0e99e58f-f194-4d91-a1ac-301719124754, queue=awx_private_queue)
[2021-04-24 10:54:42,786: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0e99e58f-f194-4d91-a1ac-301719124754, queue=awx_private_queue)
2021-04-24 10:54:42,991 DEBUG awx.main.dispatch delivered 0e99e58f-f194-4d91-a1ac-301719124754 to worker[182] qsize 0
2021-04-24 10:54:42,992 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8c6afc2f-6c50-4486-81a4-66718afa86c2, queue=awx_private_queue)
[2021-04-24 10:54:42,992: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8c6afc2f-6c50-4486-81a4-66718afa86c2, queue=awx_private_queue)
2021-04-24 10:54:42,993 DEBUG awx.main.dispatch task 0e99e58f-f194-4d91-a1ac-301719124754 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:54:42,994 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:54:43,011 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:54:43,025 DEBUG awx.main.dispatch delivered 8c6afc2f-6c50-4486-81a4-66718afa86c2 to worker[183] qsize 0
2021-04-24 10:54:43,027 DEBUG awx.main.dispatch task 8c6afc2f-6c50-4486-81a4-66718afa86c2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:54:43,034 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:54:43,037 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:54:13.021782+00:00
2021-04-24 10:54:43,044 DEBUG awx.main.dispatch task 0e99e58f-f194-4d91-a1ac-301719124754 is finished
2021-04-24 10:54:43,048 DEBUG awx.main.dispatch task 8c6afc2f-6c50-4486-81a4-66718afa86c2 is finished
RESULT 2
OKREADY
2021-04-24 10:55:02,795 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(94726533-6c4c-4f02-af13-09a2c16995e5, queue=awx_private_queue)
[2021-04-24 10:55:02,795: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(94726533-6c4c-4f02-af13-09a2c16995e5, queue=awx_private_queue)
2021-04-24 10:55:02,810 DEBUG awx.main.dispatch delivered 94726533-6c4c-4f02-af13-09a2c16995e5 to worker[183] qsize 0
2021-04-24 10:55:02,813 DEBUG awx.main.dispatch task 94726533-6c4c-4f02-af13-09a2c16995e5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:55:02,814 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:55:02,831 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:55:02,858 DEBUG awx.main.dispatch task 94726533-6c4c-4f02-af13-09a2c16995e5 is finished
2021-04-24 10:55:12,992 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(be80c96a-cae9-4968-8e2f-7bec93817bde, queue=awx_private_queue)
[2021-04-24 10:55:12,992: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(be80c96a-cae9-4968-8e2f-7bec93817bde, queue=awx_private_queue)
2021-04-24 10:55:13,008 DEBUG awx.main.dispatch delivered be80c96a-cae9-4968-8e2f-7bec93817bde to worker[180] qsize 0
2021-04-24 10:55:13,010 DEBUG awx.main.dispatch task be80c96a-cae9-4968-8e2f-7bec93817bde starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:55:13,024 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:55:13,030 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:54:43.035040+00:00
2021-04-24 10:55:13,042 DEBUG awx.main.dispatch task be80c96a-cae9-4968-8e2f-7bec93817bde is finished
2021-04-24 10:55:22,801 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3145f4e7-f1ee-4e9c-a92e-fc4091293bb4, queue=awx_private_queue)
[2021-04-24 10:55:22,801: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3145f4e7-f1ee-4e9c-a92e-fc4091293bb4, queue=awx_private_queue)
2021-04-24 10:55:22,824 DEBUG awx.main.dispatch delivered 3145f4e7-f1ee-4e9c-a92e-fc4091293bb4 to worker[180] qsize 0
2021-04-24 10:55:22,831 DEBUG awx.main.dispatch task 3145f4e7-f1ee-4e9c-a92e-fc4091293bb4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:55:22,835 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:55:23,310 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:55:23,540 DEBUG awx.main.dispatch task 3145f4e7-f1ee-4e9c-a92e-fc4091293bb4 is finished
2021-04-24 10:55:42,423 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a3077b94-9ed9-4053-abc8-ebd4ffacfbce, queue=dev-awx-0)
[2021-04-24 10:55:42,423: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a3077b94-9ed9-4053-abc8-ebd4ffacfbce, queue=dev-awx-0)
2021-04-24 10:55:42,567 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(7d26abd4-0f9e-44de-9cbb-2b5822c50d54, queue=dev-awx-0)
[2021-04-24 10:55:42,567: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(7d26abd4-0f9e-44de-9cbb-2b5822c50d54, queue=dev-awx-0)
2021-04-24 10:55:45,310 DEBUG awx.main.dispatch delivered a3077b94-9ed9-4053-abc8-ebd4ffacfbce to worker[180] qsize 0
2021-04-24 10:55:45,519 DEBUG awx.main.dispatch task a3077b94-9ed9-4053-abc8-ebd4ffacfbce starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 10:55:45,521 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 10:55:45,522 DEBUG awx.main.dispatch delivered 7d26abd4-0f9e-44de-9cbb-2b5822c50d54 to worker[177] qsize 0
2021-04-24 10:55:45,526 DEBUG awx.main.dispatch task 7d26abd4-0f9e-44de-9cbb-2b5822c50d54 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 10:55:42,796 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(38813705-41ce-4a5f-8bd4-485ab66a6b17, queue=awx_private_queue)
[2021-04-24 10:55:42,796: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(38813705-41ce-4a5f-8bd4-485ab66a6b17, queue=awx_private_queue)
2021-04-24 10:55:46,887 DEBUG awx.main.dispatch delivered 38813705-41ce-4a5f-8bd4-485ab66a6b17 to worker[183] qsize 0
2021-04-24 10:55:46,888 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2ccbbac5-5fe4-4579-b4f9-bc46fae097f1, queue=awx_private_queue)
[2021-04-24 10:55:46,888: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2ccbbac5-5fe4-4579-b4f9-bc46fae097f1, queue=awx_private_queue)
2021-04-24 10:55:46,891 DEBUG awx.main.dispatch task 38813705-41ce-4a5f-8bd4-485ab66a6b17 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:55:46,895 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:55:46,903 DEBUG awx.main.dispatch delivered 2ccbbac5-5fe4-4579-b4f9-bc46fae097f1 to worker[182] qsize 0
2021-04-24 10:55:46,906 DEBUG awx.main.dispatch task 2ccbbac5-5fe4-4579-b4f9-bc46fae097f1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:55:47,181 DEBUG awx.main.dispatch task 7d26abd4-0f9e-44de-9cbb-2b5822c50d54 is finished
2021-04-24 10:55:47,198 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:55:47,842 DEBUG awx.main.dispatch task a3077b94-9ed9-4053-abc8-ebd4ffacfbce is finished
2021-04-24 10:55:47,958 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:55:47,980 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:55:13.024501+00:00
2021-04-24 10:55:48,032 DEBUG awx.main.dispatch task 2ccbbac5-5fe4-4579-b4f9-bc46fae097f1 is finished
2021-04-24 10:55:51,888 DEBUG awx.main.dispatch task 38813705-41ce-4a5f-8bd4-485ab66a6b17 is finished
RESULT 2
OKREADY
2021-04-24 10:56:02,807 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d2a80b62-128c-46b4-98d2-0e1dfb5d7488, queue=awx_private_queue)
[2021-04-24 10:56:02,807: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d2a80b62-128c-46b4-98d2-0e1dfb5d7488, queue=awx_private_queue)
2021-04-24 10:56:02,831 DEBUG awx.main.dispatch delivered d2a80b62-128c-46b4-98d2-0e1dfb5d7488 to worker[183] qsize 0
2021-04-24 10:56:02,839 DEBUG awx.main.dispatch task d2a80b62-128c-46b4-98d2-0e1dfb5d7488 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:56:02,845 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:56:03,652 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:56:03,730 DEBUG awx.main.dispatch task d2a80b62-128c-46b4-98d2-0e1dfb5d7488 is finished
2021-04-24 10:56:16,890 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e13d1a27-3a70-4274-b11c-5056517ad294, queue=awx_private_queue)
[2021-04-24 10:56:16,890: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e13d1a27-3a70-4274-b11c-5056517ad294, queue=awx_private_queue)
2021-04-24 10:56:16,904 DEBUG awx.main.dispatch delivered e13d1a27-3a70-4274-b11c-5056517ad294 to worker[183] qsize 0
2021-04-24 10:56:16,906 DEBUG awx.main.dispatch task e13d1a27-3a70-4274-b11c-5056517ad294 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:56:19,051 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:56:19,132 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:55:47.958624+00:00
2021-04-24 10:56:20,209 DEBUG awx.main.dispatch task e13d1a27-3a70-4274-b11c-5056517ad294 is finished
2021-04-24 10:56:22,804 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b13ab861-c780-4c89-b024-1b559e7bd56f, queue=awx_private_queue)
[2021-04-24 10:56:22,804: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b13ab861-c780-4c89-b024-1b559e7bd56f, queue=awx_private_queue)
2021-04-24 10:56:22,818 DEBUG awx.main.dispatch delivered b13ab861-c780-4c89-b024-1b559e7bd56f to worker[177] qsize 0
2021-04-24 10:56:22,821 DEBUG awx.main.dispatch task b13ab861-c780-4c89-b024-1b559e7bd56f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:56:22,824 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:56:25,295 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:56:25,645 DEBUG awx.main.dispatch task b13ab861-c780-4c89-b024-1b559e7bd56f is finished
2021-04-24 10:56:42,440 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(9e68d1ef-f289-4334-bd21-37f50a70f835, queue=dev-awx-0)
[2021-04-24 10:56:42,440: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(9e68d1ef-f289-4334-bd21-37f50a70f835, queue=dev-awx-0)
2021-04-24 10:56:42,567 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2a7b3efe-d805-4fac-89c4-581ffaaa81df, queue=dev-awx-0)
[2021-04-24 10:56:42,567: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2a7b3efe-d805-4fac-89c4-581ffaaa81df, queue=dev-awx-0)
2021-04-24 10:56:42,804 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e58e94ea-d08b-4443-b3f5-7ea04a669490, queue=awx_private_queue)
[2021-04-24 10:56:42,804: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e58e94ea-d08b-4443-b3f5-7ea04a669490, queue=awx_private_queue)
2021-04-24 10:56:43,021 DEBUG awx.main.dispatch delivered 9e68d1ef-f289-4334-bd21-37f50a70f835 to worker[177] qsize 0
2021-04-24 10:56:43,026 DEBUG awx.main.dispatch delivered 2a7b3efe-d805-4fac-89c4-581ffaaa81df to worker[182] qsize 0
2021-04-24 10:56:43,028 DEBUG awx.main.dispatch delivered e58e94ea-d08b-4443-b3f5-7ea04a669490 to worker[180] qsize 0
2021-04-24 10:56:43,028 DEBUG awx.main.dispatch task 9e68d1ef-f289-4334-bd21-37f50a70f835 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 10:56:43,033 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 10:56:43,030 DEBUG awx.main.dispatch task 2a7b3efe-d805-4fac-89c4-581ffaaa81df starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 10:56:43,032 DEBUG awx.main.dispatch task e58e94ea-d08b-4443-b3f5-7ea04a669490 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:56:43,036 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:56:43,700 DEBUG awx.main.dispatch task 2a7b3efe-d805-4fac-89c4-581ffaaa81df is finished
2021-04-24 10:56:43,721 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:56:43,745 DEBUG awx.main.dispatch task 9e68d1ef-f289-4334-bd21-37f50a70f835 is finished
2021-04-24 10:56:43,815 DEBUG awx.main.dispatch task e58e94ea-d08b-4443-b3f5-7ea04a669490 is finished
2021-04-24 10:56:46,889 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b349306b-42e2-42fb-82ed-77ba94961afe, queue=awx_private_queue)
[2021-04-24 10:56:46,889: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b349306b-42e2-42fb-82ed-77ba94961afe, queue=awx_private_queue)
2021-04-24 10:56:48,550 DEBUG awx.main.dispatch delivered b349306b-42e2-42fb-82ed-77ba94961afe to worker[182] qsize 0
2021-04-24 10:56:48,746 DEBUG awx.main.dispatch task b349306b-42e2-42fb-82ed-77ba94961afe starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:56:48,779 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:56:48,784 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:56:19.051871+00:00
2021-04-24 10:56:48,837 DEBUG awx.main.dispatch task b349306b-42e2-42fb-82ed-77ba94961afe is finished
RESULT 2
OKREADY
2021-04-24 10:57:02,808 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(92e2b3a6-ac06-4415-a9f5-e70a99bad87e, queue=awx_private_queue)
[2021-04-24 10:57:02,808: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(92e2b3a6-ac06-4415-a9f5-e70a99bad87e, queue=awx_private_queue)
2021-04-24 10:57:02,824 DEBUG awx.main.dispatch delivered 92e2b3a6-ac06-4415-a9f5-e70a99bad87e to worker[177] qsize 0
2021-04-24 10:57:02,826 DEBUG awx.main.dispatch task 92e2b3a6-ac06-4415-a9f5-e70a99bad87e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:57:02,828 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:57:02,852 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:57:02,888 DEBUG awx.main.dispatch task 92e2b3a6-ac06-4415-a9f5-e70a99bad87e is finished
2021-04-24 10:57:16,889 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0609cac2-2e3c-4d64-8459-bcf17ec997ad, queue=awx_private_queue)
[2021-04-24 10:57:16,889: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0609cac2-2e3c-4d64-8459-bcf17ec997ad, queue=awx_private_queue)
2021-04-24 10:57:16,901 DEBUG awx.main.dispatch delivered 0609cac2-2e3c-4d64-8459-bcf17ec997ad to worker[183] qsize 0
2021-04-24 10:57:16,905 DEBUG awx.main.dispatch task 0609cac2-2e3c-4d64-8459-bcf17ec997ad starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:57:16,931 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:57:16,960 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:56:48.780084+00:00
2021-04-24 10:57:16,986 DEBUG awx.main.dispatch task 0609cac2-2e3c-4d64-8459-bcf17ec997ad is finished
2021-04-24 10:57:22,809 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(291c65c6-202f-41cc-bd45-3ee3469ee41b, queue=awx_private_queue)
[2021-04-24 10:57:22,809: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(291c65c6-202f-41cc-bd45-3ee3469ee41b, queue=awx_private_queue)
2021-04-24 10:57:22,830 DEBUG awx.main.dispatch delivered 291c65c6-202f-41cc-bd45-3ee3469ee41b to worker[183] qsize 0
2021-04-24 10:57:22,831 DEBUG awx.main.dispatch task 291c65c6-202f-41cc-bd45-3ee3469ee41b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:57:22,833 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:57:22,869 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:57:22,918 DEBUG awx.main.dispatch task 291c65c6-202f-41cc-bd45-3ee3469ee41b is finished
2021-04-24 10:57:42,445 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6dea6d5a-6bdd-4c0b-af02-7fe2ce00b673, queue=dev-awx-0)
[2021-04-24 10:57:42,445: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6dea6d5a-6bdd-4c0b-af02-7fe2ce00b673, queue=dev-awx-0)
2021-04-24 10:57:42,543 DEBUG awx.main.dispatch delivered 6dea6d5a-6bdd-4c0b-af02-7fe2ce00b673 to worker[182] qsize 0
2021-04-24 10:57:42,546 DEBUG awx.main.dispatch task 6dea6d5a-6bdd-4c0b-af02-7fe2ce00b673 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 10:57:42,547 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 10:57:42,567 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(452314a7-2490-43ea-bb94-c81efca6eec1, queue=dev-awx-0)
[2021-04-24 10:57:42,567: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(452314a7-2490-43ea-bb94-c81efca6eec1, queue=dev-awx-0)
2021-04-24 10:57:42,580 DEBUG awx.main.dispatch delivered 452314a7-2490-43ea-bb94-c81efca6eec1 to worker[183] qsize 0
2021-04-24 10:57:42,582 DEBUG awx.main.dispatch task 452314a7-2490-43ea-bb94-c81efca6eec1 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 10:57:42,592 DEBUG awx.main.dispatch task 6dea6d5a-6bdd-4c0b-af02-7fe2ce00b673 is finished
2021-04-24 10:57:42,597 DEBUG awx.main.dispatch task 452314a7-2490-43ea-bb94-c81efca6eec1 is finished
2021-04-24 10:57:42,808 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(555fc826-e347-4646-a5fa-73fda7ac5b51, queue=awx_private_queue)
[2021-04-24 10:57:42,808: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(555fc826-e347-4646-a5fa-73fda7ac5b51, queue=awx_private_queue)
2021-04-24 10:57:42,823 DEBUG awx.main.dispatch delivered 555fc826-e347-4646-a5fa-73fda7ac5b51 to worker[180] qsize 0
2021-04-24 10:57:42,824 DEBUG awx.main.dispatch task 555fc826-e347-4646-a5fa-73fda7ac5b51 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:57:42,826 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:57:43,023 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:57:43,087 DEBUG awx.main.dispatch task 555fc826-e347-4646-a5fa-73fda7ac5b51 is finished
2021-04-24 10:57:46,889 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f39d5b86-8a8c-4007-b1b9-515571802574, queue=awx_private_queue)
[2021-04-24 10:57:46,889: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f39d5b86-8a8c-4007-b1b9-515571802574, queue=awx_private_queue)
2021-04-24 10:57:46,905 DEBUG awx.main.dispatch delivered f39d5b86-8a8c-4007-b1b9-515571802574 to worker[180] qsize 0
2021-04-24 10:57:46,906 DEBUG awx.main.dispatch task f39d5b86-8a8c-4007-b1b9-515571802574 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:57:46,970 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:57:46,974 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:57:16.931452+00:00
2021-04-24 10:57:47,017 DEBUG awx.main.dispatch task f39d5b86-8a8c-4007-b1b9-515571802574 is finished
RESULT 2
OKREADY
2021-04-24 10:58:02,815 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9dd16edb-3383-4f70-843b-6812789897e0, queue=awx_private_queue)
[2021-04-24 10:58:02,815: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9dd16edb-3383-4f70-843b-6812789897e0, queue=awx_private_queue)
2021-04-24 10:58:02,866 DEBUG awx.main.dispatch delivered 9dd16edb-3383-4f70-843b-6812789897e0 to worker[183] qsize 0
2021-04-24 10:58:03,248 DEBUG awx.main.dispatch task 9dd16edb-3383-4f70-843b-6812789897e0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:58:03,250 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:58:03,290 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:58:03,334 DEBUG awx.main.dispatch task 9dd16edb-3383-4f70-843b-6812789897e0 is finished
2021-04-24 10:58:16,895 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1384dcc0-5e2b-4915-bc28-9a6f64914909, queue=awx_private_queue)
[2021-04-24 10:58:16,895: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1384dcc0-5e2b-4915-bc28-9a6f64914909, queue=awx_private_queue)
2021-04-24 10:58:16,920 DEBUG awx.main.dispatch delivered 1384dcc0-5e2b-4915-bc28-9a6f64914909 to worker[182] qsize 0
2021-04-24 10:58:16,926 DEBUG awx.main.dispatch task 1384dcc0-5e2b-4915-bc28-9a6f64914909 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:58:17,620 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:58:17,679 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:57:46.970445+00:00
2021-04-24 10:58:17,755 DEBUG awx.main.dispatch task 1384dcc0-5e2b-4915-bc28-9a6f64914909 is finished
2021-04-24 10:58:22,816 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d4eb9199-b817-4094-969e-2e99fefeeb12, queue=awx_private_queue)
[2021-04-24 10:58:22,816: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d4eb9199-b817-4094-969e-2e99fefeeb12, queue=awx_private_queue)
2021-04-24 10:58:22,833 DEBUG awx.main.dispatch delivered d4eb9199-b817-4094-969e-2e99fefeeb12 to worker[177] qsize 0
2021-04-24 10:58:22,838 DEBUG awx.main.dispatch task d4eb9199-b817-4094-969e-2e99fefeeb12 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:58:22,844 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:58:36,838 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:58:38,598 DEBUG awx.main.dispatch task d4eb9199-b817-4094-969e-2e99fefeeb12 is finished
2021-04-24 10:58:42,457 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(59c24dc8-21c5-42b4-8061-0dd4cca5ce5c, queue=dev-awx-0)
[2021-04-24 10:58:42,457: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(59c24dc8-21c5-42b4-8061-0dd4cca5ce5c, queue=dev-awx-0)
2021-04-24 10:58:42,567 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f5ca7a0d-65bf-4768-99af-c29c0dd6e058, queue=dev-awx-0)
[2021-04-24 10:58:42,567: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f5ca7a0d-65bf-4768-99af-c29c0dd6e058, queue=dev-awx-0)
2021-04-24 10:58:42,815 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c8c87cbd-b400-4e76-b135-6e1bd1dac69b, queue=awx_private_queue)
[2021-04-24 10:58:42,815: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c8c87cbd-b400-4e76-b135-6e1bd1dac69b, queue=awx_private_queue)
2021-04-24 10:58:46,890 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(292ff721-2565-4c04-bade-eedf1819ebbc, queue=awx_private_queue)
[2021-04-24 10:58:46,890: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(292ff721-2565-4c04-bade-eedf1819ebbc, queue=awx_private_queue)
2021-04-24 10:58:50,238 DEBUG awx.main.dispatch delivered 59c24dc8-21c5-42b4-8061-0dd4cca5ce5c to worker[182] qsize 0
2021-04-24 10:58:50,240 DEBUG awx.main.dispatch delivered f5ca7a0d-65bf-4768-99af-c29c0dd6e058 to worker[177] qsize 0
2021-04-24 10:58:50,240 DEBUG awx.main.dispatch task 59c24dc8-21c5-42b4-8061-0dd4cca5ce5c starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 10:58:50,242 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 10:58:50,242 DEBUG awx.main.dispatch delivered c8c87cbd-b400-4e76-b135-6e1bd1dac69b to worker[183] qsize 0
2021-04-24 10:58:50,242 DEBUG awx.main.dispatch task f5ca7a0d-65bf-4768-99af-c29c0dd6e058 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 10:58:50,243 DEBUG awx.main.dispatch delivered 292ff721-2565-4c04-bade-eedf1819ebbc to worker[180] qsize 0
2021-04-24 10:58:50,244 DEBUG awx.main.dispatch task c8c87cbd-b400-4e76-b135-6e1bd1dac69b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:58:50,249 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:58:50,248 DEBUG awx.main.dispatch task 292ff721-2565-4c04-bade-eedf1819ebbc starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:58:56,920 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:58:56,953 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:58:17.621287+00:00
2021-04-24 10:58:57,418 DEBUG awx.main.dispatch task 292ff721-2565-4c04-bade-eedf1819ebbc is finished
2021-04-24 10:58:58,605 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:58:59,093 DEBUG awx.main.dispatch task f5ca7a0d-65bf-4768-99af-c29c0dd6e058 is finished
2021-04-24 10:58:59,553 DEBUG awx.main.dispatch task 59c24dc8-21c5-42b4-8061-0dd4cca5ce5c is finished
2021-04-24 10:59:00,138 DEBUG awx.main.dispatch task c8c87cbd-b400-4e76-b135-6e1bd1dac69b is finished
RESULT 2
OKREADY
2021-04-24 10:59:02,818 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(316f9d8d-98df-4554-95dc-ca9a05676429, queue=awx_private_queue)
[2021-04-24 10:59:02,818: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(316f9d8d-98df-4554-95dc-ca9a05676429, queue=awx_private_queue)
2021-04-24 10:59:02,833 DEBUG awx.main.dispatch delivered 316f9d8d-98df-4554-95dc-ca9a05676429 to worker[180] qsize 0
2021-04-24 10:59:02,835 DEBUG awx.main.dispatch task 316f9d8d-98df-4554-95dc-ca9a05676429 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:59:02,836 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:59:03,094 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:59:05,994 DEBUG awx.main.dispatch task 316f9d8d-98df-4554-95dc-ca9a05676429 is finished
2021-04-24 10:59:16,896 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e759b758-8840-4d76-8af7-0be551185bda, queue=awx_private_queue)
[2021-04-24 10:59:16,896: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e759b758-8840-4d76-8af7-0be551185bda, queue=awx_private_queue)
2021-04-24 10:59:22,819 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2c8330f7-00b3-4066-a2c4-11c9bbd670cf, queue=awx_private_queue)
[2021-04-24 10:59:22,819: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2c8330f7-00b3-4066-a2c4-11c9bbd670cf, queue=awx_private_queue)
2021-04-24 10:59:16,919 DEBUG awx.main.dispatch delivered e759b758-8840-4d76-8af7-0be551185bda to worker[183] qsize 0
2021-04-24 10:59:25,540 DEBUG awx.main.dispatch delivered 2c8330f7-00b3-4066-a2c4-11c9bbd670cf to worker[180] qsize 0
2021-04-24 10:59:25,542 DEBUG awx.main.dispatch task 2c8330f7-00b3-4066-a2c4-11c9bbd670cf starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 10:59:25,543 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 10:59:25,542 DEBUG awx.main.dispatch task e759b758-8840-4d76-8af7-0be551185bda starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 10:59:25,763 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 10:59:25,802 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:58:56.920679+00:00
2021-04-24 10:59:25,818 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 10:59:28,342 DEBUG awx.main.dispatch task e759b758-8840-4d76-8af7-0be551185bda is finished
2021-04-24 10:59:39,116 DEBUG awx.main.dispatch task 2c8330f7-00b3-4066-a2c4-11c9bbd670cf is finished
2021-04-24 10:59:42,460 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(26c851a3-a024-46dd-901a-346b53eff718, queue=dev-awx-0)
[2021-04-24 10:59:42,460: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(26c851a3-a024-46dd-901a-346b53eff718, queue=dev-awx-0)
2021-04-24 10:59:42,567 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(45e37cce-e3d1-4743-9161-3145c067dbce, queue=dev-awx-0)
[2021-04-24 10:59:42,567: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(45e37cce-e3d1-4743-9161-3145c067dbce, queue=dev-awx-0)
2021-04-24 10:59:42,819 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(12ffafac-c2dc-43f5-b826-2aafd9f85987, queue=awx_private_queue)
[2021-04-24 10:59:42,819: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(12ffafac-c2dc-43f5-b826-2aafd9f85987, queue=awx_private_queue)
2021-04-24 10:59:46,899 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(94455264-f382-44e3-8606-389cd0bbede1, queue=awx_private_queue)
[2021-04-24 10:59:46,899: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(94455264-f382-44e3-8606-389cd0bbede1, queue=awx_private_queue)
RESULT 2
OKREADY
2021-04-24 11:00:02,838 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d8546c94-3c51-4681-a8b5-dc885e32bf9d, queue=awx_private_queue)
[2021-04-24 11:00:02,838: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d8546c94-3c51-4681-a8b5-dc885e32bf9d, queue=awx_private_queue)
2021-04-24 11:00:07,143 DEBUG awx.main.dispatch delivered 26c851a3-a024-46dd-901a-346b53eff718 to worker[182] qsize 0
2021-04-24 11:00:07,151 DEBUG awx.main.dispatch delivered 45e37cce-e3d1-4743-9161-3145c067dbce to worker[180] qsize 0
2021-04-24 11:00:07,153 DEBUG awx.main.dispatch delivered 12ffafac-c2dc-43f5-b826-2aafd9f85987 to worker[177] qsize 0
2021-04-24 11:00:07,156 DEBUG awx.main.dispatch delivered 94455264-f382-44e3-8606-389cd0bbede1 to worker[183] qsize 0
2021-04-24 11:00:07,154 DEBUG awx.main.dispatch task 45e37cce-e3d1-4743-9161-3145c067dbce starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:00:07,151 DEBUG awx.main.dispatch task 26c851a3-a024-46dd-901a-346b53eff718 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 11:00:07,162 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 11:00:07,158 DEBUG awx.main.dispatch task 12ffafac-c2dc-43f5-b826-2aafd9f85987 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:00:07,160 DEBUG awx.main.dispatch task 94455264-f382-44e3-8606-389cd0bbede1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:00:07,166 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:00:07,179 WARNING awx.main.dispatch scaling up worker pid:192
2021-04-24 11:00:07,185 DEBUG awx.main.dispatch delivered d8546c94-3c51-4681-a8b5-dc885e32bf9d to worker[192] qsize 0
2021-04-24 11:00:07,191 DEBUG awx.main.dispatch task d8546c94-3c51-4681-a8b5-dc885e32bf9d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:00:07,193 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:00:08,308 DEBUG awx.main.dispatch task 45e37cce-e3d1-4743-9161-3145c067dbce is finished
2021-04-24 11:00:09,322 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:00:09,607 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 10:59:25.763524+00:00
2021-04-24 11:00:11,223 DEBUG awx.main.dispatch task 94455264-f382-44e3-8606-389cd0bbede1 is finished
2021-04-24 11:00:11,387 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:00:11,578 DEBUG awx.main.dispatch task 26c851a3-a024-46dd-901a-346b53eff718 is finished
2021-04-24 11:00:12,097 DEBUG awx.main.dispatch task d8546c94-3c51-4681-a8b5-dc885e32bf9d is finished
2021-04-24 11:00:16,900 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(24c33a1f-89d5-4bac-b117-b17b706ac5e4, queue=awx_private_queue)
[2021-04-24 11:00:16,900: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(24c33a1f-89d5-4bac-b117-b17b706ac5e4, queue=awx_private_queue)
2021-04-24 11:00:16,919 DEBUG awx.main.dispatch delivered 24c33a1f-89d5-4bac-b117-b17b706ac5e4 to worker[182] qsize 0
2021-04-24 11:00:16,921 DEBUG awx.main.dispatch task 24c33a1f-89d5-4bac-b117-b17b706ac5e4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:00:17,750 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:00:22,832 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7cbf9eb7-dd08-421f-b33d-d76d1d9574ed, queue=awx_private_queue)
[2021-04-24 11:00:22,832: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7cbf9eb7-dd08-421f-b33d-d76d1d9574ed, queue=awx_private_queue)
2021-04-24 11:00:22,847 DEBUG awx.main.dispatch delivered 7cbf9eb7-dd08-421f-b33d-d76d1d9574ed to worker[192] qsize 0
2021-04-24 11:00:22,849 DEBUG awx.main.dispatch task 7cbf9eb7-dd08-421f-b33d-d76d1d9574ed starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:00:22,851 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:00:26,911 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-24 11:00:26,912 DEBUG awx.main.dispatch task 7cbf9eb7-dd08-421f-b33d-d76d1d9574ed is finished
2021-04-24 11:00:28,919 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:00:28,947 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:00:09.322454+00:00
2021-04-24 11:00:29,231 DEBUG awx.main.dispatch task 12ffafac-c2dc-43f5-b826-2aafd9f85987 is finished
2021-04-24 11:00:30,348 DEBUG awx.main.dispatch task 24c33a1f-89d5-4bac-b117-b17b706ac5e4 is finished
2021-04-24 11:00:42,470 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(4231e5ff-7b3a-42cf-a248-9b667c76f8d1, queue=dev-awx-0)
[2021-04-24 11:00:42,470: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(4231e5ff-7b3a-42cf-a248-9b667c76f8d1, queue=dev-awx-0)
2021-04-24 11:00:42,485 WARNING awx.main.dispatch scaling down worker pid:177
2021-04-24 11:00:42,488 WARNING awx.main.dispatch worker exiting gracefully pid:177
2021-04-24 11:00:42,567 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(31bda605-b543-458e-8ee3-961c7dd283d2, queue=dev-awx-0)
[2021-04-24 11:00:42,567: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(31bda605-b543-458e-8ee3-961c7dd283d2, queue=dev-awx-0)
RESULT 2
OKREADY
2021-04-24 11:00:42,832 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(eac56b3e-de1b-4716-8646-55e5ce1c5025, queue=awx_private_queue)
[2021-04-24 11:00:42,832: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(eac56b3e-de1b-4716-8646-55e5ce1c5025, queue=awx_private_queue)
2021-04-24 11:01:10,599 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a7b10e8b-6057-422d-8695-84cb61b0ccab, queue=awx_private_queue)
[2021-04-24 11:01:10,599: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a7b10e8b-6057-422d-8695-84cb61b0ccab, queue=awx_private_queue)
2021-04-24 11:01:10,615 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(46166552-bf64-444e-aa2a-6d23186f8f99, queue=awx_private_queue)
[2021-04-24 11:01:10,615: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(46166552-bf64-444e-aa2a-6d23186f8f99, queue=awx_private_queue)
2021-04-24 11:01:12,603 DEBUG awx.main.dispatch delivered 4231e5ff-7b3a-42cf-a248-9b667c76f8d1 to worker[183] qsize 0
2021-04-24 11:01:12,606 DEBUG awx.main.dispatch delivered 31bda605-b543-458e-8ee3-961c7dd283d2 to worker[192] qsize 0
2021-04-24 11:01:12,606 DEBUG awx.main.dispatch task 4231e5ff-7b3a-42cf-a248-9b667c76f8d1 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 11:01:12,608 DEBUG awx.main.dispatch delivered eac56b3e-de1b-4716-8646-55e5ce1c5025 to worker[180] qsize 0
2021-04-24 11:01:12,609 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 11:01:12,609 DEBUG awx.main.dispatch task 31bda605-b543-458e-8ee3-961c7dd283d2 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:01:12,610 DEBUG awx.main.dispatch delivered a7b10e8b-6057-422d-8695-84cb61b0ccab to worker[182] qsize 0
2021-04-24 11:01:12,610 DEBUG awx.main.dispatch task eac56b3e-de1b-4716-8646-55e5ce1c5025 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:01:12,614 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:01:12,612 DEBUG awx.main.dispatch task a7b10e8b-6057-422d-8695-84cb61b0ccab starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:01:12,626 WARNING awx.main.dispatch scaling up worker pid:195
2021-04-24 11:01:12,630 DEBUG awx.main.dispatch delivered 46166552-bf64-444e-aa2a-6d23186f8f99 to worker[195] qsize 0
2021-04-24 11:01:12,635 DEBUG awx.main.dispatch task 46166552-bf64-444e-aa2a-6d23186f8f99 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:01:12,637 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:01:12,877 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:01:12,962 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:00:28.919220+00:00
2021-04-24 11:01:13,025 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:01:13,044 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-24 11:01:13,044 DEBUG awx.main.dispatch task eac56b3e-de1b-4716-8646-55e5ce1c5025 is finished
2021-04-24 11:01:13,419 DEBUG awx.main.dispatch task a7b10e8b-6057-422d-8695-84cb61b0ccab is finished
2021-04-24 11:01:13,418 DEBUG awx.main.dispatch task 31bda605-b543-458e-8ee3-961c7dd283d2 is finished
2021-04-24 11:01:13,619 DEBUG awx.main.dispatch task 4231e5ff-7b3a-42cf-a248-9b667c76f8d1 is finished
2021-04-24 11:01:13,676 DEBUG awx.main.dispatch task 46166552-bf64-444e-aa2a-6d23186f8f99 is finished
2021-04-24 11:01:30,616 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0b6f03a6-5569-45c6-a63b-96cca12a14c4, queue=awx_private_queue)
[2021-04-24 11:01:30,616: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0b6f03a6-5569-45c6-a63b-96cca12a14c4, queue=awx_private_queue)
2021-04-24 11:01:30,634 DEBUG awx.main.dispatch delivered 0b6f03a6-5569-45c6-a63b-96cca12a14c4 to worker[195] qsize 0
2021-04-24 11:01:30,637 DEBUG awx.main.dispatch task 0b6f03a6-5569-45c6-a63b-96cca12a14c4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:01:30,640 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:01:33,521 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:01:33,877 DEBUG awx.main.dispatch task 0b6f03a6-5569-45c6-a63b-96cca12a14c4 is finished
2021-04-24 11:01:40,599 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b06d15b8-9eea-4946-88f9-448570d3e2b0, queue=awx_private_queue)
[2021-04-24 11:01:40,599: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b06d15b8-9eea-4946-88f9-448570d3e2b0, queue=awx_private_queue)
2021-04-24 11:01:40,616 DEBUG awx.main.dispatch delivered b06d15b8-9eea-4946-88f9-448570d3e2b0 to worker[180] qsize 0
2021-04-24 11:01:40,618 DEBUG awx.main.dispatch task b06d15b8-9eea-4946-88f9-448570d3e2b0 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:01:42,470 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(9dbf50dd-24a4-4d93-a100-2b76d1123f35, queue=dev-awx-0)
[2021-04-24 11:01:42,470: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(9dbf50dd-24a4-4d93-a100-2b76d1123f35, queue=dev-awx-0)
2021-04-24 11:01:42,482 WARNING awx.main.dispatch scaling down worker pid:182
2021-04-24 11:01:42,484 WARNING awx.main.dispatch worker exiting gracefully pid:182
2021-04-24 11:01:42,568 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(56a4963c-6fdc-43a9-8f33-d1ffa3891e0c, queue=dev-awx-0)
[2021-04-24 11:01:42,568: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(56a4963c-6fdc-43a9-8f33-d1ffa3891e0c, queue=dev-awx-0)
2021-04-24 11:01:44,512 DEBUG awx.main.dispatch delivered 9dbf50dd-24a4-4d93-a100-2b76d1123f35 to worker[195] qsize 0
2021-04-24 11:01:44,514 DEBUG awx.main.dispatch task 9dbf50dd-24a4-4d93-a100-2b76d1123f35 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 11:01:44,515 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 11:01:44,514 DEBUG awx.main.dispatch delivered 56a4963c-6fdc-43a9-8f33-d1ffa3891e0c to worker[192] qsize 0
2021-04-24 11:01:44,516 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:01:44,517 DEBUG awx.main.dispatch task 56a4963c-6fdc-43a9-8f33-d1ffa3891e0c starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:01:44,526 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:01:12.877661+00:00
2021-04-24 11:01:44,627 DEBUG awx.main.dispatch task b06d15b8-9eea-4946-88f9-448570d3e2b0 is finished
2021-04-24 11:01:44,647 DEBUG awx.main.dispatch task 9dbf50dd-24a4-4d93-a100-2b76d1123f35 is finished
2021-04-24 11:01:44,681 DEBUG awx.main.dispatch task 56a4963c-6fdc-43a9-8f33-d1ffa3891e0c is finished
2021-04-24 11:01:50,615 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ab1d2186-9f0a-4cc4-9cf6-1d21e4acb6da, queue=awx_private_queue)
[2021-04-24 11:01:50,615: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ab1d2186-9f0a-4cc4-9cf6-1d21e4acb6da, queue=awx_private_queue)
2021-04-24 11:01:50,630 DEBUG awx.main.dispatch delivered ab1d2186-9f0a-4cc4-9cf6-1d21e4acb6da to worker[183] qsize 0
2021-04-24 11:01:50,632 DEBUG awx.main.dispatch task ab1d2186-9f0a-4cc4-9cf6-1d21e4acb6da starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:01:50,634 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:01:53,877 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:01:53,971 DEBUG awx.main.dispatch task ab1d2186-9f0a-4cc4-9cf6-1d21e4acb6da is finished
RESULT 2
OKREADY
2021-04-24 11:02:10,603 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e5546aa0-7ac4-4520-9d9a-f43d6f6e0973, queue=awx_private_queue)
[2021-04-24 11:02:10,603: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e5546aa0-7ac4-4520-9d9a-f43d6f6e0973, queue=awx_private_queue)
2021-04-24 11:02:12,600 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(33452a4a-6644-43d6-90cb-d28b4b54dae0, queue=awx_private_queue)
[2021-04-24 11:02:12,600: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(33452a4a-6644-43d6-90cb-d28b4b54dae0, queue=awx_private_queue)
2021-04-24 11:02:12,599 DEBUG awx.main.dispatch delivered e5546aa0-7ac4-4520-9d9a-f43d6f6e0973 to worker[192] qsize 0
2021-04-24 11:02:12,603 DEBUG awx.main.dispatch task e5546aa0-7ac4-4520-9d9a-f43d6f6e0973 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:02:12,618 DEBUG awx.main.dispatch delivered 33452a4a-6644-43d6-90cb-d28b4b54dae0 to worker[195] qsize 0
2021-04-24 11:02:12,620 DEBUG awx.main.dispatch task 33452a4a-6644-43d6-90cb-d28b4b54dae0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:02:12,621 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:02:13,149 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:02:13,163 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:01:44.517123+00:00
2021-04-24 11:02:13,178 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:02:13,242 DEBUG awx.main.dispatch task e5546aa0-7ac4-4520-9d9a-f43d6f6e0973 is finished
2021-04-24 11:02:14,349 DEBUG awx.main.dispatch task 33452a4a-6644-43d6-90cb-d28b4b54dae0 is finished
2021-04-24 11:02:32,610 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f779d9f6-4624-47b2-b0c0-1947a2f7383d, queue=awx_private_queue)
[2021-04-24 11:02:32,610: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f779d9f6-4624-47b2-b0c0-1947a2f7383d, queue=awx_private_queue)
2021-04-24 11:02:32,636 DEBUG awx.main.dispatch delivered f779d9f6-4624-47b2-b0c0-1947a2f7383d to worker[180] qsize 0
2021-04-24 11:02:35,577 DEBUG awx.main.dispatch task f779d9f6-4624-47b2-b0c0-1947a2f7383d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:02:35,582 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:02:35,884 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:02:35,944 DEBUG awx.main.dispatch task f779d9f6-4624-47b2-b0c0-1947a2f7383d is finished
2021-04-24 11:02:42,470 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(4f44c758-90c5-4355-b03d-d892ccc95fcb, queue=dev-awx-0)
[2021-04-24 11:02:42,470: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(4f44c758-90c5-4355-b03d-d892ccc95fcb, queue=dev-awx-0)
2021-04-24 11:02:42,538 DEBUG awx.main.dispatch delivered 4f44c758-90c5-4355-b03d-d892ccc95fcb to worker[180] qsize 0
2021-04-24 11:02:42,540 DEBUG awx.main.dispatch task 4f44c758-90c5-4355-b03d-d892ccc95fcb starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 11:02:42,542 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 11:02:42,568 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(59827fae-06c3-4006-a052-a1e9546c00f7, queue=dev-awx-0)
[2021-04-24 11:02:42,568: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(59827fae-06c3-4006-a052-a1e9546c00f7, queue=dev-awx-0)
2021-04-24 11:02:42,589 DEBUG awx.main.dispatch delivered 59827fae-06c3-4006-a052-a1e9546c00f7 to worker[195] qsize 0
2021-04-24 11:02:42,591 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d549fd22-0e98-4d49-82d8-0de6880c98d6, queue=awx_private_queue)
[2021-04-24 11:02:42,591: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d549fd22-0e98-4d49-82d8-0de6880c98d6, queue=awx_private_queue)
2021-04-24 11:02:42,592 DEBUG awx.main.dispatch task 59827fae-06c3-4006-a052-a1e9546c00f7 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:02:42,606 DEBUG awx.main.dispatch task 4f44c758-90c5-4355-b03d-d892ccc95fcb is finished
2021-04-24 11:02:42,610 DEBUG awx.main.dispatch delivered d549fd22-0e98-4d49-82d8-0de6880c98d6 to worker[180] qsize 0
2021-04-24 11:02:42,611 DEBUG awx.main.dispatch task d549fd22-0e98-4d49-82d8-0de6880c98d6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:02:42,635 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:02:42,642 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:02:13.150324+00:00
2021-04-24 11:02:42,645 DEBUG awx.main.dispatch task 59827fae-06c3-4006-a052-a1e9546c00f7 is finished
2021-04-24 11:02:42,681 DEBUG awx.main.dispatch task d549fd22-0e98-4d49-82d8-0de6880c98d6 is finished
2021-04-24 11:02:52,609 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e1bd8f0a-8b49-4930-84f4-74587de2d8e7, queue=awx_private_queue)
[2021-04-24 11:02:52,609: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e1bd8f0a-8b49-4930-84f4-74587de2d8e7, queue=awx_private_queue)
2021-04-24 11:02:52,628 DEBUG awx.main.dispatch delivered e1bd8f0a-8b49-4930-84f4-74587de2d8e7 to worker[183] qsize 0
2021-04-24 11:02:52,632 DEBUG awx.main.dispatch task e1bd8f0a-8b49-4930-84f4-74587de2d8e7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:02:52,637 DEBUG awx.main.scheduler Running Tower task manager.
RESULT 2
OKREADY
2021-04-24 11:03:10,145 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:03:11,717 DEBUG awx.main.dispatch task e1bd8f0a-8b49-4930-84f4-74587de2d8e7 is finished
2021-04-24 11:03:12,600 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(159fc0c2-09c1-46a0-9a43-d4cb303c2012, queue=awx_private_queue)
[2021-04-24 11:03:12,600: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(159fc0c2-09c1-46a0-9a43-d4cb303c2012, queue=awx_private_queue)
2021-04-24 11:03:12,616 DEBUG awx.main.dispatch delivered 159fc0c2-09c1-46a0-9a43-d4cb303c2012 to worker[192] qsize 0
2021-04-24 11:03:12,619 DEBUG awx.main.dispatch task 159fc0c2-09c1-46a0-9a43-d4cb303c2012 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:03:12,624 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fd1a9600-5b63-46d2-9d80-154d4e5a6548, queue=awx_private_queue)
[2021-04-24 11:03:12,624: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fd1a9600-5b63-46d2-9d80-154d4e5a6548, queue=awx_private_queue)
2021-04-24 11:03:12,638 DEBUG awx.main.dispatch delivered fd1a9600-5b63-46d2-9d80-154d4e5a6548 to worker[180] qsize 0
2021-04-24 11:03:12,640 DEBUG awx.main.dispatch task fd1a9600-5b63-46d2-9d80-154d4e5a6548 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:03:12,641 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:03:15,081 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:03:15,097 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:02:42.636013+00:00
2021-04-24 11:03:15,114 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:03:15,137 DEBUG awx.main.dispatch task 159fc0c2-09c1-46a0-9a43-d4cb303c2012 is finished
2021-04-24 11:03:15,357 DEBUG awx.main.dispatch task fd1a9600-5b63-46d2-9d80-154d4e5a6548 is finished
2021-04-24 11:03:32,633 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9843887a-c4c9-47ee-8c46-abb2ad488aca, queue=awx_private_queue)
[2021-04-24 11:03:32,633: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9843887a-c4c9-47ee-8c46-abb2ad488aca, queue=awx_private_queue)
2021-04-24 11:03:32,650 DEBUG awx.main.dispatch delivered 9843887a-c4c9-47ee-8c46-abb2ad488aca to worker[183] qsize 0
2021-04-24 11:03:32,653 DEBUG awx.main.dispatch task 9843887a-c4c9-47ee-8c46-abb2ad488aca starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:03:32,654 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:03:46,063 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:03:42,470 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6639e693-395c-401f-b87b-f328b380d78d, queue=dev-awx-0)
[2021-04-24 11:03:42,470: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6639e693-395c-401f-b87b-f328b380d78d, queue=dev-awx-0)
2021-04-24 11:03:46,079 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(e45a0759-d080-435c-9861-f39d442094e0, queue=dev-awx-0)
[2021-04-24 11:03:46,079: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(e45a0759-d080-435c-9861-f39d442094e0, queue=dev-awx-0)
2021-04-24 11:03:46,098 DEBUG awx.main.dispatch task 9843887a-c4c9-47ee-8c46-abb2ad488aca is finished
2021-04-24 11:03:46,100 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e21a396e-3ffa-4765-9797-81235fd98877, queue=awx_private_queue)
[2021-04-24 11:03:46,100: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e21a396e-3ffa-4765-9797-81235fd98877, queue=awx_private_queue)
2021-04-24 11:03:46,336 DEBUG awx.main.dispatch delivered 6639e693-395c-401f-b87b-f328b380d78d to worker[195] qsize 0
2021-04-24 11:03:46,348 DEBUG awx.main.dispatch delivered e45a0759-d080-435c-9861-f39d442094e0 to worker[192] qsize 0
2021-04-24 11:03:46,350 DEBUG awx.main.dispatch delivered e21a396e-3ffa-4765-9797-81235fd98877 to worker[183] qsize 0
2021-04-24 11:03:46,350 DEBUG awx.main.dispatch task e45a0759-d080-435c-9861-f39d442094e0 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:03:46,351 DEBUG awx.main.dispatch task e21a396e-3ffa-4765-9797-81235fd98877 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:03:46,598 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:03:46,607 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:03:15.081925+00:00
2021-04-24 11:03:46,606 DEBUG awx.main.dispatch task e45a0759-d080-435c-9861-f39d442094e0 is finished
2021-04-24 11:03:46,339 DEBUG awx.main.dispatch task 6639e693-395c-401f-b87b-f328b380d78d starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 11:03:46,815 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 11:03:48,801 DEBUG awx.main.dispatch task 6639e693-395c-401f-b87b-f328b380d78d is finished
2021-04-24 11:03:52,633 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4560c0e2-8695-4c15-9146-3ed51f75ab0b, queue=awx_private_queue)
[2021-04-24 11:03:52,633: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4560c0e2-8695-4c15-9146-3ed51f75ab0b, queue=awx_private_queue)
2021-04-24 11:03:52,651 DEBUG awx.main.dispatch delivered 4560c0e2-8695-4c15-9146-3ed51f75ab0b to worker[180] qsize 0
2021-04-24 11:03:52,653 DEBUG awx.main.dispatch task 4560c0e2-8695-4c15-9146-3ed51f75ab0b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:03:52,655 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:03:53,701 DEBUG awx.main.dispatch task e21a396e-3ffa-4765-9797-81235fd98877 is finished
2021-04-24 11:04:01,476 DEBUG awx.main.scheduler Starting Scheduler
RESULT 2
OKREADY
2021-04-24 11:04:12,652 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(99da170e-87ac-4037-9bac-d53a08a1386b, queue=awx_private_queue)
[2021-04-24 11:04:12,652: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(99da170e-87ac-4037-9bac-d53a08a1386b, queue=awx_private_queue)
2021-04-24 11:04:12,684 DEBUG awx.main.dispatch delivered 99da170e-87ac-4037-9bac-d53a08a1386b to worker[195] qsize 0
2021-04-24 11:04:12,689 DEBUG awx.main.dispatch task 99da170e-87ac-4037-9bac-d53a08a1386b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:04:12,691 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:04:16,101 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2515bf67-641f-49e8-ae36-da5c962ed3b6, queue=awx_private_queue)
[2021-04-24 11:04:16,101: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2515bf67-641f-49e8-ae36-da5c962ed3b6, queue=awx_private_queue)
2021-04-24 11:04:16,116 DEBUG awx.main.dispatch delivered 2515bf67-641f-49e8-ae36-da5c962ed3b6 to worker[183] qsize 0
2021-04-24 11:04:16,120 DEBUG awx.main.dispatch task 2515bf67-641f-49e8-ae36-da5c962ed3b6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:04:21,460 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:04:21,495 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:03:46.598400+00:00
2021-04-24 11:04:21,675 DEBUG awx.main.dispatch task 2515bf67-641f-49e8-ae36-da5c962ed3b6 is finished
2021-04-24 11:04:22,516 DEBUG awx.main.dispatch task 4560c0e2-8695-4c15-9146-3ed51f75ab0b is finished
2021-04-24 11:04:24,049 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:04:25,919 DEBUG awx.main.dispatch task 99da170e-87ac-4037-9bac-d53a08a1386b is finished
2021-04-24 11:04:32,645 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1ae88de5-6064-4c6b-9987-3bcb735407e1, queue=awx_private_queue)
[2021-04-24 11:04:32,645: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1ae88de5-6064-4c6b-9987-3bcb735407e1, queue=awx_private_queue)
2021-04-24 11:04:32,661 DEBUG awx.main.dispatch delivered 1ae88de5-6064-4c6b-9987-3bcb735407e1 to worker[195] qsize 0
2021-04-24 11:04:32,665 DEBUG awx.main.dispatch task 1ae88de5-6064-4c6b-9987-3bcb735407e1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:04:32,668 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:04:36,862 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:04:36,972 DEBUG awx.main.dispatch task 1ae88de5-6064-4c6b-9987-3bcb735407e1 is finished
2021-04-24 11:04:42,347 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(28a50a5c-04e6-4385-9e4a-8c0d3f4e49d8, queue=dev-awx-0)
[2021-04-24 11:04:42,347: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(28a50a5c-04e6-4385-9e4a-8c0d3f4e49d8, queue=dev-awx-0)
2021-04-24 11:04:42,360 DEBUG awx.main.dispatch delivered 28a50a5c-04e6-4385-9e4a-8c0d3f4e49d8 to worker[183] qsize 0
2021-04-24 11:04:42,363 DEBUG awx.main.dispatch task 28a50a5c-04e6-4385-9e4a-8c0d3f4e49d8 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-04-24 11:04:42,365 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-04-24 11:04:42,470 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(0d490f7d-b9db-41d9-96c4-97dab31c6931, queue=dev-awx-0)
[2021-04-24 11:04:42,470: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(0d490f7d-b9db-41d9-96c4-97dab31c6931, queue=dev-awx-0)
2021-04-24 11:04:42,601 DEBUG awx.main.dispatch task 28a50a5c-04e6-4385-9e4a-8c0d3f4e49d8 is finished
2021-04-24 11:04:42,602 DEBUG awx.main.dispatch delivered 0d490f7d-b9db-41d9-96c4-97dab31c6931 to worker[192] qsize 0
2021-04-24 11:04:42,604 DEBUG awx.main.dispatch task 0d490f7d-b9db-41d9-96c4-97dab31c6931 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 11:04:42,609 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 11:04:42,645 DEBUG awx.main.dispatch task 0d490f7d-b9db-41d9-96c4-97dab31c6931 is finished
2021-04-24 11:04:46,079 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(e615ded2-495e-4b29-ad49-7687a2a3264f, queue=dev-awx-0)
[2021-04-24 11:04:46,079: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(e615ded2-495e-4b29-ad49-7687a2a3264f, queue=dev-awx-0)
2021-04-24 11:04:46,094 DEBUG awx.main.dispatch delivered e615ded2-495e-4b29-ad49-7687a2a3264f to worker[192] qsize 0
2021-04-24 11:04:46,096 DEBUG awx.main.dispatch task e615ded2-495e-4b29-ad49-7687a2a3264f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:04:46,101 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(618e9c58-59af-4a8b-b21f-d28fe312af39, queue=awx_private_queue)
[2021-04-24 11:04:46,101: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(618e9c58-59af-4a8b-b21f-d28fe312af39, queue=awx_private_queue)
2021-04-24 11:04:46,114 DEBUG awx.main.dispatch delivered 618e9c58-59af-4a8b-b21f-d28fe312af39 to worker[183] qsize 0
2021-04-24 11:04:46,116 DEBUG awx.main.dispatch task 618e9c58-59af-4a8b-b21f-d28fe312af39 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:04:47,328 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:04:47,339 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:04:21.460800+00:00
2021-04-24 11:04:47,374 DEBUG awx.main.dispatch task e615ded2-495e-4b29-ad49-7687a2a3264f is finished
2021-04-24 11:04:47,379 DEBUG awx.main.dispatch task 618e9c58-59af-4a8b-b21f-d28fe312af39 is finished
RESULT 2
OKREADY
2021-04-24 11:04:52,645 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(16ab0fb1-e33d-4728-9e4f-98553fbf7220, queue=awx_private_queue)
[2021-04-24 11:04:52,645: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(16ab0fb1-e33d-4728-9e4f-98553fbf7220, queue=awx_private_queue)
2021-04-24 11:05:49,308 DEBUG awx.main.dispatch delivered 16ab0fb1-e33d-4728-9e4f-98553fbf7220 to worker[192] qsize 0
2021-04-24 11:05:49,312 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4c09940d-fd86-4c04-bfb6-e8c1e3be55dc, queue=awx_private_queue)
[2021-04-24 11:05:49,312: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4c09940d-fd86-4c04-bfb6-e8c1e3be55dc, queue=awx_private_queue)
2021-04-24 11:05:49,314 DEBUG awx.main.dispatch task 16ab0fb1-e33d-4728-9e4f-98553fbf7220 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:05:49,318 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:05:49,327 DEBUG awx.main.dispatch delivered 4c09940d-fd86-4c04-bfb6-e8c1e3be55dc to worker[180] qsize 0
2021-04-24 11:05:49,330 DEBUG awx.main.dispatch task 4c09940d-fd86-4c04-bfb6-e8c1e3be55dc starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:05:49,331 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(31dd1800-a2d5-402d-bf5c-219edcbf888a, queue=dev-awx-0)
[2021-04-24 11:05:49,331: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(31dd1800-a2d5-402d-bf5c-219edcbf888a, queue=dev-awx-0)
2021-04-24 11:05:49,357 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(97f02c5f-b1ff-4b7f-8040-9c4962e33546, queue=dev-awx-0)
[2021-04-24 11:05:49,357: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(97f02c5f-b1ff-4b7f-8040-9c4962e33546, queue=dev-awx-0)
2021-04-24 11:05:49,379 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(391a666a-18e8-4598-ba10-144c853dff67, queue=awx_private_queue)
[2021-04-24 11:05:49,379: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(391a666a-18e8-4598-ba10-144c853dff67, queue=awx_private_queue)
RESULT 2
OKREADY
2021-04-24 11:06:09,397 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(475ba0d1-1232-4fe9-b2c1-42fdedf1d1b4, queue=awx_private_queue)
[2021-04-24 11:06:09,397: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(475ba0d1-1232-4fe9-b2c1-42fdedf1d1b4, queue=awx_private_queue)
2021-04-24 11:06:19,315 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(badda97d-3488-4933-9488-827c67c1b3e2, queue=awx_private_queue)
[2021-04-24 11:06:19,315: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(badda97d-3488-4933-9488-827c67c1b3e2, queue=awx_private_queue)
2021-04-24 11:06:29,391 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8ca2adb3-41ec-49eb-91bd-f3d763544c3c, queue=awx_private_queue)
[2021-04-24 11:06:29,391: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8ca2adb3-41ec-49eb-91bd-f3d763544c3c, queue=awx_private_queue)
2021-04-24 11:06:48,575 DEBUG awx.main.dispatch delivered 31dd1800-a2d5-402d-bf5c-219edcbf888a to worker[195] qsize 0
2021-04-24 11:06:48,591 DEBUG awx.main.dispatch delivered 97f02c5f-b1ff-4b7f-8040-9c4962e33546 to worker[183] qsize 0
2021-04-24 11:06:48,613 WARNING awx.main.dispatch scaling up worker pid:198
2021-04-24 11:06:48,619 DEBUG awx.main.dispatch delivered 391a666a-18e8-4598-ba10-144c853dff67 to worker[198] qsize 0
2021-04-24 11:06:48,627 DEBUG awx.main.dispatch task 391a666a-18e8-4598-ba10-144c853dff67 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:06:48,629 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:06:48,630 WARNING awx.main.dispatch scaling up worker pid:200
2021-04-24 11:06:48,635 DEBUG awx.main.dispatch delivered 475ba0d1-1232-4fe9-b2c1-42fdedf1d1b4 to worker[200] qsize 0
2021-04-24 11:06:48,640 DEBUG awx.main.dispatch task 475ba0d1-1232-4fe9-b2c1-42fdedf1d1b4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:06:48,642 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:06:48,646 WARNING awx.main.dispatch scaling up worker pid:202
2021-04-24 11:06:48,651 DEBUG awx.main.dispatch delivered badda97d-3488-4933-9488-827c67c1b3e2 to worker[202] qsize 0
2021-04-24 11:06:48,655 DEBUG awx.main.dispatch task badda97d-3488-4933-9488-827c67c1b3e2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:06:48,661 WARNING awx.main.dispatch scaling up worker pid:204
2021-04-24 11:06:48,666 DEBUG awx.main.dispatch delivered 8ca2adb3-41ec-49eb-91bd-f3d763544c3c to worker[204] qsize 0
2021-04-24 11:06:48,671 DEBUG awx.main.dispatch task 8ca2adb3-41ec-49eb-91bd-f3d763544c3c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:06:48,673 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:06:49,325 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(49b58d1a-abcd-4464-a566-e775df62571b, queue=awx_private_queue)
[2021-04-24 11:06:49,325: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(49b58d1a-abcd-4464-a566-e775df62571b, queue=awx_private_queue)
2021-04-24 11:06:49,342 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5b613578-b676-4f42-9178-8ac2915a3b86, queue=dev-awx-0)
[2021-04-24 11:06:49,342: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5b613578-b676-4f42-9178-8ac2915a3b86, queue=dev-awx-0)
2021-04-24 11:06:49,359 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d49a8f77-01cf-4178-8b26-794ae0b47424, queue=dev-awx-0)
[2021-04-24 11:06:49,359: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d49a8f77-01cf-4178-8b26-794ae0b47424, queue=dev-awx-0)
2021-04-24 11:06:49,392 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4a18741f-e8cf-4a1e-861d-e656d45e762d, queue=awx_private_queue)
[2021-04-24 11:06:49,392: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4a18741f-e8cf-4a1e-861d-e656d45e762d, queue=awx_private_queue)
2021-04-24 11:06:58,909 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:06:59,665 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-04-24 11:06:48,592 DEBUG awx.main.dispatch task 31dd1800-a2d5-402d-bf5c-219edcbf888a starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 11:06:59,885 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 11:06:48,594 DEBUG awx.main.dispatch task 97f02c5f-b1ff-4b7f-8040-9c4962e33546 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:06:59,913 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:06:59,930 DEBUG awx.main.dispatch task 97f02c5f-b1ff-4b7f-8040-9c4962e33546 is finished
2021-04-24 11:06:59,666 DEBUG awx.main.dispatch task 4c09940d-fd86-4c04-bfb6-e8c1e3be55dc is finished
2021-04-24 11:06:59,961 WARNING awx.main.tasks Rejoining the cluster as instance dev-awx-0.
2021-04-24 11:06:59,961 DEBUG awx.main.dispatch task 31dd1800-a2d5-402d-bf5c-219edcbf888a is finished
2021-04-24 11:07:00,014 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:04:47.328590+00:00
2021-04-24 11:07:00,100 DEBUG awx.main.dispatch task badda97d-3488-4933-9488-827c67c1b3e2 is finished
RESULT 2
OKREADY
2021-04-24 11:07:00,150 DEBUG awx.main.dispatch task 16ab0fb1-e33d-4728-9e4f-98553fbf7220 is finished
2021-04-24 11:07:00,181 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:07:03,131 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-24 11:07:03,131 DEBUG awx.main.dispatch task 391a666a-18e8-4598-ba10-144c853dff67 is finished
2021-04-24 11:07:09,392 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8a6ffc61-40e4-4918-83d6-1876f5a58d6b, queue=awx_private_queue)
[2021-04-24 11:07:09,392: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8a6ffc61-40e4-4918-83d6-1876f5a58d6b, queue=awx_private_queue)
2021-04-24 11:07:19,325 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4b7e3725-5a29-4cf4-a34a-ecb3bc415d07, queue=awx_private_queue)
[2021-04-24 11:07:19,325: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4b7e3725-5a29-4cf4-a34a-ecb3bc415d07, queue=awx_private_queue)
2021-04-24 11:07:20,706 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-24 11:07:20,706 DEBUG awx.main.dispatch task 475ba0d1-1232-4fe9-b2c1-42fdedf1d1b4 is finished
2021-04-24 11:06:49,348 WARNING awx.main.dispatch scaling up worker pid:206
2021-04-24 11:07:26,791 DEBUG awx.main.dispatch delivered 49b58d1a-abcd-4464-a566-e775df62571b to worker[180] qsize 0
2021-04-24 11:07:26,794 WARNING awx.main.dispatch scaling down worker pid:183
2021-04-24 11:07:26,793 DEBUG awx.main.dispatch task 49b58d1a-abcd-4464-a566-e775df62571b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:07:26,796 WARNING awx.main.dispatch scaling down worker pid:192
2021-04-24 11:07:26,796 WARNING awx.main.dispatch worker exiting gracefully pid:183
2021-04-24 11:07:26,798 WARNING awx.main.dispatch scaling down worker pid:195
2021-04-24 11:07:26,799 WARNING awx.main.dispatch scaling down worker pid:198
2021-04-24 11:07:26,801 WARNING awx.main.dispatch scaling down worker pid:200
2021-04-24 11:07:26,801 WARNING awx.main.dispatch worker exiting gracefully pid:195
2021-04-24 11:07:26,802 WARNING awx.main.dispatch worker exiting gracefully pid:198
2021-04-24 11:07:26,799 WARNING awx.main.dispatch worker exiting gracefully pid:192
2021-04-24 11:07:26,803 WARNING awx.main.dispatch worker exiting gracefully pid:200
2021-04-24 11:07:27,406 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:07:27,449 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:06:58.909287+00:00
2021-04-24 11:07:27,518 DEBUG awx.main.dispatch task 49b58d1a-abcd-4464-a566-e775df62571b is finished
2021-04-24 11:07:27,521 DEBUG awx.main.dispatch delivered 5b613578-b676-4f42-9178-8ac2915a3b86 to worker[202] qsize 0
2021-04-24 11:07:27,524 DEBUG awx.main.dispatch task 5b613578-b676-4f42-9178-8ac2915a3b86 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 11:07:27,526 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 11:07:27,525 DEBUG awx.main.dispatch delivered d49a8f77-01cf-4178-8b26-794ae0b47424 to worker[206] qsize 0
2021-04-24 11:07:27,529 DEBUG awx.main.dispatch delivered 4a18741f-e8cf-4a1e-861d-e656d45e762d to worker[180] qsize 0
2021-04-24 11:07:27,531 DEBUG awx.main.dispatch task 4a18741f-e8cf-4a1e-861d-e656d45e762d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:07:27,532 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:07:27,530 DEBUG awx.main.dispatch task d49a8f77-01cf-4178-8b26-794ae0b47424 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:07:27,542 WARNING awx.main.dispatch scaling up worker pid:211
2021-04-24 11:07:27,550 DEBUG awx.main.dispatch delivered 8a6ffc61-40e4-4918-83d6-1876f5a58d6b to worker[211] qsize 0
2021-04-24 11:07:27,556 DEBUG awx.main.dispatch task 8a6ffc61-40e4-4918-83d6-1876f5a58d6b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:07:27,558 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:07:27,564 WARNING awx.main.dispatch scaling up worker pid:213
2021-04-24 11:07:27,570 DEBUG awx.main.dispatch delivered 4b7e3725-5a29-4cf4-a34a-ecb3bc415d07 to worker[213] qsize 0
2021-04-24 11:07:27,575 DEBUG awx.main.dispatch task 4b7e3725-5a29-4cf4-a34a-ecb3bc415d07 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:07:29,391 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bc26f70a-1b08-48fb-aa80-e7c47da69edb, queue=awx_private_queue)
[2021-04-24 11:07:29,391: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bc26f70a-1b08-48fb-aa80-e7c47da69edb, queue=awx_private_queue)
2021-04-24 11:07:29,412 WARNING awx.main.dispatch scaling up worker pid:215
2021-04-24 11:07:29,416 DEBUG awx.main.dispatch delivered bc26f70a-1b08-48fb-aa80-e7c47da69edb to worker[215] qsize 0
2021-04-24 11:07:29,421 DEBUG awx.main.dispatch task bc26f70a-1b08-48fb-aa80-e7c47da69edb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:07:29,424 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:07:41,561 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-24 11:07:41,562 DEBUG awx.main.dispatch task 4a18741f-e8cf-4a1e-861d-e656d45e762d is finished
2021-04-24 11:07:49,335 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(71aa174d-77a9-48ac-8d86-5e3ef569f090, queue=awx_private_queue)
[2021-04-24 11:07:49,335: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(71aa174d-77a9-48ac-8d86-5e3ef569f090, queue=awx_private_queue)
2021-04-24 11:07:49,356 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(43c28a1b-7bdb-408f-a25a-4e28f7b1efc2, queue=dev-awx-0)
[2021-04-24 11:07:49,356: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(43c28a1b-7bdb-408f-a25a-4e28f7b1efc2, queue=dev-awx-0)
2021-04-24 11:07:49,356 DEBUG awx.main.dispatch delivered 71aa174d-77a9-48ac-8d86-5e3ef569f090 to worker[180] qsize 0
2021-04-24 11:07:49,360 DEBUG awx.main.dispatch task 71aa174d-77a9-48ac-8d86-5e3ef569f090 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:07:49,368 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(87faeb30-4f44-4a48-a56e-e1e942f702f1, queue=dev-awx-0)
[2021-04-24 11:07:49,368: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(87faeb30-4f44-4a48-a56e-e1e942f702f1, queue=dev-awx-0)
2021-04-24 11:07:49,391 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3d0d7138-86b6-4c87-8fd0-b32dbe77a82b, queue=awx_private_queue)
[2021-04-24 11:07:49,391: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3d0d7138-86b6-4c87-8fd0-b32dbe77a82b, queue=awx_private_queue)
RESULT 2
OKREADY
2021-04-24 11:08:04,402 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:08:09,405 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4b9545c6-c549-40df-b59a-de9dea320e91, queue=awx_private_queue)
[2021-04-24 11:08:09,405: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4b9545c6-c549-40df-b59a-de9dea320e91, queue=awx_private_queue)
2021-04-24 11:08:11,663 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:07:27.406502+00:00
2021-04-24 11:08:14,730 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-04-24 11:08:14,731 DEBUG awx.main.dispatch task 71aa174d-77a9-48ac-8d86-5e3ef569f090 is finished
2021-04-24 11:08:14,758 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-24 11:08:14,759 DEBUG awx.main.dispatch task bc26f70a-1b08-48fb-aa80-e7c47da69edb is finished
2021-04-24 11:08:14,913 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-24 11:08:14,914 DEBUG awx.main.dispatch task 8a6ffc61-40e4-4918-83d6-1876f5a58d6b is finished
2021-04-24 11:08:15,786 DEBUG awx.main.dispatch delivered 43c28a1b-7bdb-408f-a25a-4e28f7b1efc2 to worker[211] qsize 0
2021-04-24 11:08:15,790 DEBUG awx.main.dispatch delivered 87faeb30-4f44-4a48-a56e-e1e942f702f1 to worker[215] qsize 0
2021-04-24 11:08:15,790 DEBUG awx.main.dispatch task 43c28a1b-7bdb-408f-a25a-4e28f7b1efc2 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 11:08:15,791 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 11:08:15,791 DEBUG awx.main.dispatch delivered 3d0d7138-86b6-4c87-8fd0-b32dbe77a82b to worker[180] qsize 0
2021-04-24 11:08:15,791 DEBUG awx.main.dispatch task 87faeb30-4f44-4a48-a56e-e1e942f702f1 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:08:15,793 DEBUG awx.main.dispatch task 3d0d7138-86b6-4c87-8fd0-b32dbe77a82b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:08:15,804 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:08:15,800 WARNING awx.main.dispatch scaling up worker pid:219
2021-04-24 11:08:15,805 DEBUG awx.main.dispatch delivered 4b9545c6-c549-40df-b59a-de9dea320e91 to worker[219] qsize 0
2021-04-24 11:08:15,811 DEBUG awx.main.dispatch task 4b9545c6-c549-40df-b59a-de9dea320e91 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:08:15,814 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:08:17,339 DEBUG awx.main.dispatch task 5b613578-b676-4f42-9178-8ac2915a3b86 is finished
2021-04-24 11:08:17,494 DEBUG awx.main.dispatch task 4b7e3725-5a29-4cf4-a34a-ecb3bc415d07 is finished
2021-04-24 11:08:19,335 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e42795ab-e5ea-46fd-a07b-fbca636bb36b, queue=awx_private_queue)
[2021-04-24 11:08:19,335: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e42795ab-e5ea-46fd-a07b-fbca636bb36b, queue=awx_private_queue)
2021-04-24 11:08:19,356 DEBUG awx.main.dispatch delivered e42795ab-e5ea-46fd-a07b-fbca636bb36b to worker[213] qsize 0
2021-04-24 11:08:19,358 DEBUG awx.main.dispatch task e42795ab-e5ea-46fd-a07b-fbca636bb36b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:08:35,381 DEBUG awx.main.dispatch task 8ca2adb3-41ec-49eb-91bd-f3d763544c3c is finished
2021-04-24 11:08:36,686 DEBUG awx.main.dispatch task d49a8f77-01cf-4178-8b26-794ae0b47424 is finished
RESULT 2
OKREADY
2021-04-24 11:09:10,714 DEBUG awx.main.dispatch task 87faeb30-4f44-4a48-a56e-e1e942f702f1 is finished
2021-04-24 11:09:11,161 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:08:29,403 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(57b14975-59f7-4226-ba18-e24ee83bdb89, queue=awx_private_queue)
[2021-04-24 11:08:29,403: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(57b14975-59f7-4226-ba18-e24ee83bdb89, queue=awx_private_queue)
2021-04-24 11:09:13,304 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(262641a3-4062-4fcf-8e04-1871fedf8fb7, queue=awx_private_queue)
[2021-04-24 11:09:13,304: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(262641a3-4062-4fcf-8e04-1871fedf8fb7, queue=awx_private_queue)
2021-04-24 11:09:13,304 DEBUG awx.main.dispatch delivered 57b14975-59f7-4226-ba18-e24ee83bdb89 to worker[204] qsize 0
2021-04-24 11:09:13,311 DEBUG awx.main.dispatch task 57b14975-59f7-4226-ba18-e24ee83bdb89 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:09:13,316 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:09:13,316 DEBUG awx.main.dispatch delivered 262641a3-4062-4fcf-8e04-1871fedf8fb7 to worker[202] qsize 0
2021-04-24 11:09:13,317 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8486ee53-ff29-4af2-97f7-baf9f03c6a6b, queue=dev-awx-0)
[2021-04-24 11:09:13,317: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8486ee53-ff29-4af2-97f7-baf9f03c6a6b, queue=dev-awx-0)
2021-04-24 11:09:13,319 DEBUG awx.main.dispatch task 262641a3-4062-4fcf-8e04-1871fedf8fb7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:09:13,333 WARNING awx.main.dispatch scaling down worker pid:206
2021-04-24 11:09:13,334 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(19902940-60c3-4afa-b1af-3aa0ada7759c, queue=dev-awx-0)
[2021-04-24 11:09:13,334: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(19902940-60c3-4afa-b1af-3aa0ada7759c, queue=dev-awx-0)
2021-04-24 11:09:13,335 WARNING awx.main.dispatch worker exiting gracefully pid:206
2021-04-24 11:09:13,334 WARNING awx.main.dispatch scaling down worker pid:215
2021-04-24 11:09:13,347 WARNING awx.main.dispatch worker exiting gracefully pid:215
2021-04-24 11:09:13,350 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d16de0d4-04e5-4875-81bd-f0741789e6e1, queue=awx_private_queue)
[2021-04-24 11:09:13,350: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d16de0d4-04e5-4875-81bd-f0741789e6e1, queue=awx_private_queue)
2021-04-24 11:09:23,362 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:08:04.403710+00:00
2021-04-24 11:09:25,634 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:09:26,034 DEBUG awx.main.dispatch task 43c28a1b-7bdb-408f-a25a-4e28f7b1efc2 is finished
2021-04-24 11:09:26,542 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-24 11:09:26,542 DEBUG awx.main.dispatch task 3d0d7138-86b6-4c87-8fd0-b32dbe77a82b is finished
2021-04-24 11:09:29,502 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-04-24 11:09:29,502 DEBUG awx.main.dispatch task 262641a3-4062-4fcf-8e04-1871fedf8fb7 is finished
2021-04-24 11:09:33,351 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(08d1a9a9-8aae-4135-ad0d-0d2d06746827, queue=awx_private_queue)
[2021-04-24 11:09:33,351: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(08d1a9a9-8aae-4135-ad0d-0d2d06746827, queue=awx_private_queue)
2021-04-24 11:09:43,304 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9548ddc0-a164-4a21-8a6a-ba4a573886b2, queue=awx_private_queue)
[2021-04-24 11:09:43,304: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9548ddc0-a164-4a21-8a6a-ba4a573886b2, queue=awx_private_queue)
2021-04-24 11:09:35,796 DEBUG awx.main.dispatch task 4b9545c6-c549-40df-b59a-de9dea320e91 is finished
2021-04-24 11:09:46,649 DEBUG awx.main.dispatch delivered 8486ee53-ff29-4af2-97f7-baf9f03c6a6b to worker[180] qsize 0
2021-04-24 11:09:46,650 DEBUG awx.main.dispatch delivered 19902940-60c3-4afa-b1af-3aa0ada7759c to worker[211] qsize 0
2021-04-24 11:09:46,650 DEBUG awx.main.dispatch task 8486ee53-ff29-4af2-97f7-baf9f03c6a6b starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 11:09:46,652 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 11:09:46,652 DEBUG awx.main.dispatch delivered d16de0d4-04e5-4875-81bd-f0741789e6e1 to worker[202] qsize 0
2021-04-24 11:09:46,653 DEBUG awx.main.dispatch task 19902940-60c3-4afa-b1af-3aa0ada7759c starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:09:46,654 DEBUG awx.main.dispatch task d16de0d4-04e5-4875-81bd-f0741789e6e1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:09:46,656 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:09:46,654 DEBUG awx.main.dispatch delivered 08d1a9a9-8aae-4135-ad0d-0d2d06746827 to worker[219] qsize 0
2021-04-24 11:09:46,658 DEBUG awx.main.dispatch task 08d1a9a9-8aae-4135-ad0d-0d2d06746827 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:09:46,659 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:09:46,676 WARNING awx.main.dispatch scaling up worker pid:225
2021-04-24 11:09:46,683 DEBUG awx.main.dispatch delivered 9548ddc0-a164-4a21-8a6a-ba4a573886b2 to worker[225] qsize 0
2021-04-24 11:09:46,687 DEBUG awx.main.dispatch task 9548ddc0-a164-4a21-8a6a-ba4a573886b2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:09:53,351 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a589f779-139b-4604-a7c4-7c870ecc74dc, queue=awx_private_queue)
[2021-04-24 11:09:53,351: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a589f779-139b-4604-a7c4-7c870ecc74dc, queue=awx_private_queue)
2021-04-24 11:09:53,379 WARNING awx.main.dispatch scaling up worker pid:227
2021-04-24 11:09:53,386 DEBUG awx.main.dispatch delivered a589f779-139b-4604-a7c4-7c870ecc74dc to worker[227] qsize 0
2021-04-24 11:09:53,394 DEBUG awx.main.dispatch task a589f779-139b-4604-a7c4-7c870ecc74dc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:09:53,397 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:09:53,749 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:09:35,771 DEBUG awx.main.dispatch task e42795ab-e5ea-46fd-a07b-fbca636bb36b is finished
2021-04-24 11:09:54,194 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:09:54,206 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:09:11.161642+00:00
2021-04-24 11:09:54,244 DEBUG awx.main.dispatch task 19902940-60c3-4afa-b1af-3aa0ada7759c is finished
2021-04-24 11:09:54,314 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-04-24 11:09:54,314 DEBUG awx.main.dispatch task d16de0d4-04e5-4875-81bd-f0741789e6e1 is finished
2021-04-24 11:09:54,361 DEBUG awx.main.dispatch task 57b14975-59f7-4226-ba18-e24ee83bdb89 is finished
2021-04-24 11:09:54,805 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:10:01,275 DEBUG awx.main.dispatch task 08d1a9a9-8aae-4135-ad0d-0d2d06746827 is finished
RESULT 2
OKREADY
2021-04-24 11:10:01,277 DEBUG awx.main.dispatch task 9548ddc0-a164-4a21-8a6a-ba4a573886b2 is finished
2021-04-24 11:10:01,878 DEBUG awx.main.dispatch task 8486ee53-ff29-4af2-97f7-baf9f03c6a6b is finished
2021-04-24 11:10:04,087 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:10:04,244 DEBUG awx.main.dispatch task a589f779-139b-4604-a7c4-7c870ecc74dc is finished
2021-04-24 11:10:13,324 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(033719f4-e372-44e8-900f-290a1a60b78e, queue=dev-awx-0)
[2021-04-24 11:10:13,324: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(033719f4-e372-44e8-900f-290a1a60b78e, queue=dev-awx-0)
2021-04-24 11:10:13,356 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(291469e3-7e74-4178-b3de-8badb59a665d, queue=awx_private_queue)
[2021-04-24 11:10:13,356: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(291469e3-7e74-4178-b3de-8badb59a665d, queue=awx_private_queue)
2021-04-24 11:10:13,356 WARNING awx.main.dispatch scaling down worker pid:180
2021-04-24 11:10:13,360 WARNING awx.main.dispatch scaling down worker pid:202
2021-04-24 11:10:13,362 WARNING awx.main.dispatch worker exiting gracefully pid:202
2021-04-24 11:10:13,363 WARNING awx.main.dispatch scaling down worker pid:204
2021-04-24 11:10:13,366 WARNING awx.main.dispatch scaling down worker pid:211
2021-04-24 11:10:13,368 WARNING awx.main.dispatch worker exiting gracefully pid:211
2021-04-24 11:10:13,362 WARNING awx.main.dispatch worker exiting gracefully pid:180
2021-04-24 11:10:13,374 WARNING awx.main.dispatch worker exiting gracefully pid:204
2021-04-24 11:10:13,393 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(55c7256e-3dd5-4521-aff4-8f2bd561dab4, queue=dev-awx-0)
[2021-04-24 11:10:13,393: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(55c7256e-3dd5-4521-aff4-8f2bd561dab4, queue=dev-awx-0)
2021-04-24 11:10:13,415 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(93d15e98-b541-46f4-a660-7fc50bd6a852, queue=awx_private_queue)
[2021-04-24 11:10:13,415: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(93d15e98-b541-46f4-a660-7fc50bd6a852, queue=awx_private_queue)
2021-04-24 11:10:15,004 DEBUG awx.main.dispatch delivered 033719f4-e372-44e8-900f-290a1a60b78e to worker[213] qsize 0
2021-04-24 11:10:15,008 DEBUG awx.main.dispatch delivered 291469e3-7e74-4178-b3de-8badb59a665d to worker[227] qsize 0
2021-04-24 11:10:15,010 DEBUG awx.main.dispatch delivered 55c7256e-3dd5-4521-aff4-8f2bd561dab4 to worker[225] qsize 0
2021-04-24 11:10:15,013 DEBUG awx.main.dispatch delivered 93d15e98-b541-46f4-a660-7fc50bd6a852 to worker[219] qsize 0
2021-04-24 11:10:15,010 DEBUG awx.main.dispatch task 033719f4-e372-44e8-900f-290a1a60b78e starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 11:10:15,015 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 11:10:15,011 DEBUG awx.main.dispatch task 291469e3-7e74-4178-b3de-8badb59a665d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:10:15,014 DEBUG awx.main.dispatch task 55c7256e-3dd5-4521-aff4-8f2bd561dab4 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:10:15,017 DEBUG awx.main.dispatch task 93d15e98-b541-46f4-a660-7fc50bd6a852 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:10:15,020 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:10:15,063 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:10:15,067 DEBUG awx.main.dispatch task 55c7256e-3dd5-4521-aff4-8f2bd561dab4 is finished
2021-04-24 11:10:15,067 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:09:54.195192+00:00
2021-04-24 11:10:15,072 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:10:15,222 DEBUG awx.main.dispatch task 033719f4-e372-44e8-900f-290a1a60b78e is finished
2021-04-24 11:10:16,720 DEBUG awx.main.dispatch task 93d15e98-b541-46f4-a660-7fc50bd6a852 is finished
2021-04-24 11:10:19,007 DEBUG awx.main.dispatch task 291469e3-7e74-4178-b3de-8badb59a665d is finished
2021-04-24 11:10:33,427 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(38046e24-6c59-4473-86c4-12e29150be99, queue=awx_private_queue)
[2021-04-24 11:10:33,427: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(38046e24-6c59-4473-86c4-12e29150be99, queue=awx_private_queue)
2021-04-24 11:10:33,451 DEBUG awx.main.dispatch delivered 38046e24-6c59-4473-86c4-12e29150be99 to worker[225] qsize 0
2021-04-24 11:10:33,457 DEBUG awx.main.dispatch task 38046e24-6c59-4473-86c4-12e29150be99 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:10:33,459 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:10:43,356 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(94e5d9bb-2bd8-44c6-967e-959950f5f84c, queue=awx_private_queue)
[2021-04-24 11:10:43,356: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(94e5d9bb-2bd8-44c6-967e-959950f5f84c, queue=awx_private_queue)
2021-04-24 11:10:43,380 DEBUG awx.main.dispatch delivered 94e5d9bb-2bd8-44c6-967e-959950f5f84c to worker[219] qsize 0
2021-04-24 11:10:43,385 DEBUG awx.main.dispatch task 94e5d9bb-2bd8-44c6-967e-959950f5f84c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:10:53,426 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(00bf25b2-ea56-43f6-b92c-c67ff0ee63e6, queue=awx_private_queue)
[2021-04-24 11:10:53,426: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(00bf25b2-ea56-43f6-b92c-c67ff0ee63e6, queue=awx_private_queue)
2021-04-24 11:10:53,452 DEBUG awx.main.dispatch delivered 00bf25b2-ea56-43f6-b92c-c67ff0ee63e6 to worker[227] qsize 0
2021-04-24 11:10:53,458 DEBUG awx.main.dispatch task 00bf25b2-ea56-43f6-b92c-c67ff0ee63e6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:10:53,461 DEBUG awx.main.scheduler Running Tower task manager.
RESULT 2
OKREADY
2021-04-24 11:11:09,099 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:11:29,793 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:11:30,796 DEBUG awx.main.tasks Last scheduler run was: 2021-04-24 11:10:15.064135+00:00
2021-04-24 11:11:31,258 DEBUG awx.main.dispatch task 00bf25b2-ea56-43f6-b92c-c67ff0ee63e6 is finished
2021-04-24 11:11:31,550 DEBUG awx.main.scheduler Starting Scheduler
2021-04-24 11:11:32,025 DEBUG awx.main.dispatch task 38046e24-6c59-4473-86c4-12e29150be99 is finished
2021-04-24 11:11:13,330 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(25b2f15f-4764-4928-a9d9-5bb36368b8d0, queue=dev-awx-0)
[2021-04-24 11:11:13,330: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(25b2f15f-4764-4928-a9d9-5bb36368b8d0, queue=dev-awx-0)
2021-04-24 11:11:33,960 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(fee970c3-e51f-48d3-9273-fb4c010c01bd, queue=awx_private_queue)
[2021-04-24 11:11:33,960: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(fee970c3-e51f-48d3-9273-fb4c010c01bd, queue=awx_private_queue)
2021-04-24 11:11:33,981 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3c3fb692-74e4-4b61-a64c-e582fdc87862, queue=dev-awx-0)
[2021-04-24 11:11:33,981: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3c3fb692-74e4-4b61-a64c-e582fdc87862, queue=dev-awx-0)
2021-04-24 11:11:33,999 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5788b751-c899-4351-818a-5b3c7d550947, queue=awx_private_queue)
[2021-04-24 11:11:33,999: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5788b751-c899-4351-818a-5b3c7d550947, queue=awx_private_queue)
2021-04-24 11:11:37,153 DEBUG awx.main.dispatch task 94e5d9bb-2bd8-44c6-967e-959950f5f84c is finished
2021-04-24 11:11:54,015 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0283b7c3-5c6a-41a2-b5c4-4c1b227db97f, queue=awx_private_queue)
[2021-04-24 11:11:54,015: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0283b7c3-5c6a-41a2-b5c4-4c1b227db97f, queue=awx_private_queue)
RESULT 2
OKREADY
2021-04-24 11:12:03,961 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a694fcd4-400f-4a59-a38b-cfec435093ac, queue=awx_private_queue)
[2021-04-24 11:12:03,961: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a694fcd4-400f-4a59-a38b-cfec435093ac, queue=awx_private_queue)
2021-04-24 11:12:14,012 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cbd618a4-a49f-4db8-bc99-bf18a217bf3a, queue=awx_private_queue)
[2021-04-24 11:12:14,012: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cbd618a4-a49f-4db8-bc99-bf18a217bf3a, queue=awx_private_queue)
2021-04-24 11:12:14,031 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(30266461-5b04-4a45-a61e-ba0eb0aa7001, queue=dev-awx-0)
[2021-04-24 11:12:14,031: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(30266461-5b04-4a45-a61e-ba0eb0aa7001, queue=dev-awx-0)
2021-04-24 11:12:22,794 DEBUG awx.main.dispatch delivered 25b2f15f-4764-4928-a9d9-5bb36368b8d0 to worker[225] qsize 0
2021-04-24 11:12:22,803 DEBUG awx.main.dispatch delivered fee970c3-e51f-48d3-9273-fb4c010c01bd to worker[219] qsize 0
2021-04-24 11:12:22,803 DEBUG awx.main.dispatch task 25b2f15f-4764-4928-a9d9-5bb36368b8d0 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-04-24 11:12:22,806 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-04-24 11:12:22,810 DEBUG awx.main.dispatch delivered 3c3fb692-74e4-4b61-a64c-e582fdc87862 to worker[213] qsize 0
2021-04-24 11:12:22,806 DEBUG awx.main.dispatch task fee970c3-e51f-48d3-9273-fb4c010c01bd starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:12:22,811 DEBUG awx.main.dispatch delivered 5788b751-c899-4351-818a-5b3c7d550947 to worker[227] qsize 0
2021-04-24 11:12:22,820 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:12:22,820 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-04-24 11:12:22,819 DEBUG awx.main.dispatch task 5788b751-c899-4351-818a-5b3c7d550947 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:12:22,826 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:12:22,826 DEBUG awx.main.dispatch task 25b2f15f-4764-4928-a9d9-5bb36368b8d0 is finished
2021-04-24 11:12:22,825 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:12:22,814 DEBUG awx.main.dispatch task 3c3fb692-74e4-4b61-a64c-e582fdc87862 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:12:22,831 DEBUG awx.main.dispatch task fee970c3-e51f-48d3-9273-fb4c010c01bd is finished
2021-04-24 11:12:22,829 WARNING awx.main.dispatch scaling up worker pid:231
2021-04-24 11:12:22,834 DEBUG awx.main.dispatch delivered 0283b7c3-5c6a-41a2-b5c4-4c1b227db97f to worker[231] qsize 0
2021-04-24 11:12:22,839 DEBUG awx.main.dispatch delivered a694fcd4-400f-4a59-a38b-cfec435093ac to worker[219] qsize 0
2021-04-24 11:12:22,842 DEBUG awx.main.dispatch delivered cbd618a4-a49f-4db8-bc99-bf18a217bf3a to worker[225] qsize 0
2021-04-24 11:12:22,842 DEBUG awx.main.dispatch task a694fcd4-400f-4a59-a38b-cfec435093ac starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:12:22,841 DEBUG awx.main.dispatch task 0283b7c3-5c6a-41a2-b5c4-4c1b227db97f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:12:22,844 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:12:22,846 DEBUG awx.main.dispatch task cbd618a4-a49f-4db8-bc99-bf18a217bf3a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:12:22,847 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:12:22,840 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:12:22,849 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:12:22,851 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:12:22,854 DEBUG awx.main.dispatch task a694fcd4-400f-4a59-a38b-cfec435093ac is finished
2021-04-24 11:12:22,854 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:12:22,849 DEBUG awx.main.dispatch task 5788b751-c899-4351-818a-5b3c7d550947 is finished
2021-04-24 11:12:22,856 DEBUG awx.main.dispatch task cbd618a4-a49f-4db8-bc99-bf18a217bf3a is finished
2021-04-24 11:12:22,851 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140190265284160_1" does not exist
2021-04-24 11:12:22,853 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:12:22,860 DEBUG awx.main.dispatch task 0283b7c3-5c6a-41a2-b5c4-4c1b227db97f is finished
2021-04-24 11:12:22,857 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-24 11:12:22,859 DEBUG awx.main.dispatch task 3c3fb692-74e4-4b61-a64c-e582fdc87862 is finished
2021-04-24 11:12:33,978 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5a97d4f7-c315-4838-acd7-655d8e2f51d5, queue=awx_private_queue)
[2021-04-24 11:12:33,978: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5a97d4f7-c315-4838-acd7-655d8e2f51d5, queue=awx_private_queue)
2021-04-24 11:12:34,029 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3c55d008-9dcb-4e0c-bdfa-13e169b9eb4d, queue=dev-awx-0)
[2021-04-24 11:12:34,029: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3c55d008-9dcb-4e0c-bdfa-13e169b9eb4d, queue=dev-awx-0)
2021-04-24 11:12:34,029 DEBUG awx.main.dispatch delivered 5a97d4f7-c315-4838-acd7-655d8e2f51d5 to worker[219] qsize 0
2021-04-24 11:12:34,057 DEBUG awx.main.dispatch delivered 3c55d008-9dcb-4e0c-bdfa-13e169b9eb4d to worker[213] qsize 0
2021-04-24 11:12:34,069 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1adeefa8-ccfb-433d-bb7b-663727228903, queue=awx_private_queue)
[2021-04-24 11:12:34,069: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1adeefa8-ccfb-433d-bb7b-663727228903, queue=awx_private_queue)
2021-04-24 11:12:34,057 DEBUG awx.main.dispatch task 5a97d4f7-c315-4838-acd7-655d8e2f51d5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:12:34,083 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:12:34,070 DEBUG awx.main.dispatch task 3c55d008-9dcb-4e0c-bdfa-13e169b9eb4d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:12:34,086 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:12:34,091 DEBUG awx.main.dispatch task 5a97d4f7-c315-4838-acd7-655d8e2f51d5 is finished
2021-04-24 11:12:34,091 DEBUG awx.main.dispatch delivered 1adeefa8-ccfb-433d-bb7b-663727228903 to worker[225] qsize 0
2021-04-24 11:12:34,091 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140190265284160_2" does not exist
2021-04-24 11:12:34,096 DEBUG awx.main.dispatch task 3c55d008-9dcb-4e0c-bdfa-13e169b9eb4d is finished
2021-04-24 11:12:34,098 DEBUG awx.main.dispatch task 1adeefa8-ccfb-433d-bb7b-663727228903 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:12:34,125 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:12:34,127 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:12:34,133 DEBUG awx.main.dispatch task 1adeefa8-ccfb-433d-bb7b-663727228903 is finished
2021-04-24 11:12:54,082 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ccfb2dae-7877-415e-bfce-213b17648f5d, queue=awx_private_queue)
[2021-04-24 11:12:54,082: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ccfb2dae-7877-415e-bfce-213b17648f5d, queue=awx_private_queue)
2021-04-24 11:12:54,118 DEBUG awx.main.dispatch delivered ccfb2dae-7877-415e-bfce-213b17648f5d to worker[231] qsize 0
2021-04-24 11:12:54,134 DEBUG awx.main.dispatch task ccfb2dae-7877-415e-bfce-213b17648f5d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:12:54,160 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:12:54,161 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:12:54,169 DEBUG awx.main.dispatch task ccfb2dae-7877-415e-bfce-213b17648f5d is finished
RESULT 2
OKREADY
2021-04-24 11:13:03,976 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7baf66e8-f8db-41ae-8870-ee45e77a68b7, queue=awx_private_queue)
[2021-04-24 11:13:03,976: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7baf66e8-f8db-41ae-8870-ee45e77a68b7, queue=awx_private_queue)
2021-04-24 11:13:04,003 DEBUG awx.main.dispatch delivered 7baf66e8-f8db-41ae-8870-ee45e77a68b7 to worker[225] qsize 0
2021-04-24 11:13:04,015 DEBUG awx.main.dispatch task 7baf66e8-f8db-41ae-8870-ee45e77a68b7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:13:04,035 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:13:04,038 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:13:04,051 DEBUG awx.main.dispatch task 7baf66e8-f8db-41ae-8870-ee45e77a68b7 is finished
2021-04-24 11:13:14,032 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b5a4631b-d5a9-41d2-b202-942e67ae0daf, queue=dev-awx-0)
[2021-04-24 11:13:14,032: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b5a4631b-d5a9-41d2-b202-942e67ae0daf, queue=dev-awx-0)
2021-04-24 11:13:14,055 WARNING awx.main.dispatch scaling down worker pid:213
2021-04-24 11:13:14,064 WARNING awx.main.dispatch worker exiting gracefully pid:213
2021-04-24 11:13:14,069 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-24 11:13:14,080 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(dbd76470-a0db-4dbc-ba45-0e9d5cfd18f1, queue=awx_private_queue)
[2021-04-24 11:13:14,080: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(dbd76470-a0db-4dbc-ba45-0e9d5cfd18f1, queue=awx_private_queue)
2021-04-24 11:13:14,108 DEBUG awx.main.dispatch delivered dbd76470-a0db-4dbc-ba45-0e9d5cfd18f1 to worker[227] qsize 0
2021-04-24 11:13:14,117 DEBUG awx.main.dispatch task dbd76470-a0db-4dbc-ba45-0e9d5cfd18f1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:13:14,145 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:13:14,147 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:13:14,151 DEBUG awx.main.dispatch task dbd76470-a0db-4dbc-ba45-0e9d5cfd18f1 is finished
2021-04-24 11:13:33,981 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4501641a-16fa-411c-b18c-8e76ce381c53, queue=awx_private_queue)
[2021-04-24 11:13:33,981: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4501641a-16fa-411c-b18c-8e76ce381c53, queue=awx_private_queue)
2021-04-24 11:13:34,012 DEBUG awx.main.dispatch delivered 4501641a-16fa-411c-b18c-8e76ce381c53 to worker[225] qsize 0
2021-04-24 11:13:34,021 DEBUG awx.main.dispatch task 4501641a-16fa-411c-b18c-8e76ce381c53 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:13:34,035 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:13:34,028 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(efd3b192-70c9-4a52-a503-7012de9cb340, queue=dev-awx-0)
[2021-04-24 11:13:34,028: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(efd3b192-70c9-4a52-a503-7012de9cb340, queue=dev-awx-0)
2021-04-24 11:13:34,038 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:13:34,042 DEBUG awx.main.dispatch task 4501641a-16fa-411c-b18c-8e76ce381c53 is finished
2021-04-24 11:13:34,051 DEBUG awx.main.dispatch delivered efd3b192-70c9-4a52-a503-7012de9cb340 to worker[227] qsize 0
2021-04-24 11:13:34,060 DEBUG awx.main.dispatch task efd3b192-70c9-4a52-a503-7012de9cb340 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:13:34,080 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140190265284160_1" does not exist
2021-04-24 11:13:34,080 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a457bc76-06c5-496d-b5a6-0bbbb4bc61f1, queue=awx_private_queue)
[2021-04-24 11:13:34,080: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a457bc76-06c5-496d-b5a6-0bbbb4bc61f1, queue=awx_private_queue)
2021-04-24 11:13:34,087 DEBUG awx.main.dispatch task efd3b192-70c9-4a52-a503-7012de9cb340 is finished
2021-04-24 11:13:34,104 DEBUG awx.main.dispatch delivered a457bc76-06c5-496d-b5a6-0bbbb4bc61f1 to worker[231] qsize 0
2021-04-24 11:13:34,110 DEBUG awx.main.dispatch task a457bc76-06c5-496d-b5a6-0bbbb4bc61f1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:13:34,126 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:13:34,128 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:13:34,132 DEBUG awx.main.dispatch task a457bc76-06c5-496d-b5a6-0bbbb4bc61f1 is finished
2021-04-24 11:13:54,082 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8625fb7c-f07e-44b5-995f-4be872c040c8, queue=awx_private_queue)
[2021-04-24 11:13:54,082: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8625fb7c-f07e-44b5-995f-4be872c040c8, queue=awx_private_queue)
2021-04-24 11:13:54,117 DEBUG awx.main.dispatch delivered 8625fb7c-f07e-44b5-995f-4be872c040c8 to worker[227] qsize 0
2021-04-24 11:13:54,130 DEBUG awx.main.dispatch task 8625fb7c-f07e-44b5-995f-4be872c040c8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:13:54,141 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:13:54,142 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:13:54,146 DEBUG awx.main.dispatch task 8625fb7c-f07e-44b5-995f-4be872c040c8 is finished
RESULT 2
OKREADY
2021-04-24 11:14:03,983 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2603847d-1433-4800-95c6-8e54fb5c1d49, queue=awx_private_queue)
[2021-04-24 11:14:03,983: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2603847d-1433-4800-95c6-8e54fb5c1d49, queue=awx_private_queue)
2021-04-24 11:14:04,026 DEBUG awx.main.dispatch delivered 2603847d-1433-4800-95c6-8e54fb5c1d49 to worker[231] qsize 0
2021-04-24 11:14:04,033 DEBUG awx.main.dispatch task 2603847d-1433-4800-95c6-8e54fb5c1d49 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:14:04,052 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:14:04,056 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:14:04,065 DEBUG awx.main.dispatch task 2603847d-1433-4800-95c6-8e54fb5c1d49 is finished
2021-04-24 11:14:14,037 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5754a1f6-5796-4725-892a-e9fad05e38cb, queue=dev-awx-0)
[2021-04-24 11:14:14,037: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5754a1f6-5796-4725-892a-e9fad05e38cb, queue=dev-awx-0)
2021-04-24 11:14:14,082 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c054ce74-ab5c-43d4-9bdc-9b482b86361d, queue=awx_private_queue)
2021-04-24 11:14:14,085 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
[2021-04-24 11:14:14,082: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c054ce74-ab5c-43d4-9bdc-9b482b86361d, queue=awx_private_queue)
2021-04-24 11:14:14,110 DEBUG awx.main.dispatch delivered c054ce74-ab5c-43d4-9bdc-9b482b86361d to worker[219] qsize 0
2021-04-24 11:14:14,118 DEBUG awx.main.dispatch task c054ce74-ab5c-43d4-9bdc-9b482b86361d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:14:14,139 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:14:14,140 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:14:14,145 DEBUG awx.main.dispatch task c054ce74-ab5c-43d4-9bdc-9b482b86361d is finished
2021-04-24 11:14:33,990 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(34ddc060-e511-46de-a9f0-9b016ab9cbce, queue=awx_private_queue)
[2021-04-24 11:14:33,990: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(34ddc060-e511-46de-a9f0-9b016ab9cbce, queue=awx_private_queue)
2021-04-24 11:14:34,028 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2c5d059d-b60c-4708-8ce4-cb7b96f72561, queue=dev-awx-0)
[2021-04-24 11:14:34,028: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2c5d059d-b60c-4708-8ce4-cb7b96f72561, queue=dev-awx-0)
2021-04-24 11:14:34,026 DEBUG awx.main.dispatch delivered 34ddc060-e511-46de-a9f0-9b016ab9cbce to worker[219] qsize 0
2021-04-24 11:14:34,052 DEBUG awx.main.dispatch delivered 2c5d059d-b60c-4708-8ce4-cb7b96f72561 to worker[225] qsize 0
2021-04-24 11:14:34,046 DEBUG awx.main.dispatch task 34ddc060-e511-46de-a9f0-9b016ab9cbce starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:14:34,072 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:14:34,059 DEBUG awx.main.dispatch task 2c5d059d-b60c-4708-8ce4-cb7b96f72561 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:14:34,075 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:14:34,080 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140190265284160_2" does not exist
2021-04-24 11:14:34,079 DEBUG awx.main.dispatch task 34ddc060-e511-46de-a9f0-9b016ab9cbce is finished
2021-04-24 11:14:34,084 DEBUG awx.main.dispatch task 2c5d059d-b60c-4708-8ce4-cb7b96f72561 is finished
2021-04-24 11:14:34,081 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a89c598f-a12d-4e51-a560-0b73620ca5a3, queue=awx_private_queue)
[2021-04-24 11:14:34,081: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a89c598f-a12d-4e51-a560-0b73620ca5a3, queue=awx_private_queue)
2021-04-24 11:14:34,106 DEBUG awx.main.dispatch delivered a89c598f-a12d-4e51-a560-0b73620ca5a3 to worker[225] qsize 0
2021-04-24 11:14:34,112 DEBUG awx.main.dispatch task a89c598f-a12d-4e51-a560-0b73620ca5a3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:14:34,122 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:14:34,123 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:14:34,127 DEBUG awx.main.dispatch task a89c598f-a12d-4e51-a560-0b73620ca5a3 is finished
2021-04-24 11:14:42,348 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(9e9589d6-1809-400f-9133-08060581b51e, queue=dev-awx-0)
[2021-04-24 11:14:42,348: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(9e9589d6-1809-400f-9133-08060581b51e, queue=dev-awx-0)
2021-04-24 11:14:42,371 DEBUG awx.main.dispatch delivered 9e9589d6-1809-400f-9133-08060581b51e to worker[231] qsize 0
2021-04-24 11:14:42,379 DEBUG awx.main.dispatch task 9e9589d6-1809-400f-9133-08060581b51e starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-04-24 11:14:42,399 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-04-24 11:14:42,409 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 470, in awx_isolated_heartbeat
for isolated_instance in isolated_instance_qs:
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-04-24 11:14:42,413 DEBUG awx.main.dispatch task 9e9589d6-1809-400f-9133-08060581b51e is finished
2021-04-24 11:14:54,083 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a2931a9f-626e-4d98-bd38-dd21df4522e8, queue=awx_private_queue)
[2021-04-24 11:14:54,083: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a2931a9f-626e-4d98-bd38-dd21df4522e8, queue=awx_private_queue)
2021-04-24 11:14:54,108 DEBUG awx.main.dispatch delivered a2931a9f-626e-4d98-bd38-dd21df4522e8 to worker[231] qsize 0
2021-04-24 11:14:54,115 DEBUG awx.main.dispatch task a2931a9f-626e-4d98-bd38-dd21df4522e8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:14:54,128 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:14:54,129 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:14:54,134 DEBUG awx.main.dispatch task a2931a9f-626e-4d98-bd38-dd21df4522e8 is finished
RESULT 2
OKREADY
2021-04-24 11:15:03,990 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e2797325-dfcf-473a-88e0-0ba5faf2432a, queue=awx_private_queue)
[2021-04-24 11:15:03,990: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e2797325-dfcf-473a-88e0-0ba5faf2432a, queue=awx_private_queue)
2021-04-24 11:15:04,023 DEBUG awx.main.dispatch delivered e2797325-dfcf-473a-88e0-0ba5faf2432a to worker[227] qsize 0
2021-04-24 11:15:04,031 DEBUG awx.main.dispatch task e2797325-dfcf-473a-88e0-0ba5faf2432a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:15:04,056 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:15:04,058 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:15:04,062 DEBUG awx.main.dispatch task e2797325-dfcf-473a-88e0-0ba5faf2432a is finished
2021-04-24 11:15:14,032 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(003f7eb7-06d7-465c-9d04-d1a3591249c4, queue=dev-awx-0)
[2021-04-24 11:15:14,032: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(003f7eb7-06d7-465c-9d04-d1a3591249c4, queue=dev-awx-0)
2021-04-24 11:15:14,062 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-24 11:15:14,082 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(80581af5-39e8-454c-855e-79a84a8b25bc, queue=awx_private_queue)
[2021-04-24 11:15:14,082: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(80581af5-39e8-454c-855e-79a84a8b25bc, queue=awx_private_queue)
2021-04-24 11:15:14,105 DEBUG awx.main.dispatch delivered 80581af5-39e8-454c-855e-79a84a8b25bc to worker[219] qsize 0
2021-04-24 11:15:14,111 DEBUG awx.main.dispatch task 80581af5-39e8-454c-855e-79a84a8b25bc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:15:14,123 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:15:14,125 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:15:14,131 DEBUG awx.main.dispatch task 80581af5-39e8-454c-855e-79a84a8b25bc is finished
2021-04-24 11:15:34,004 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(92294876-ac64-4731-b918-68c451c0afc1, queue=awx_private_queue)
[2021-04-24 11:15:34,004: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(92294876-ac64-4731-b918-68c451c0afc1, queue=awx_private_queue)
2021-04-24 11:15:34,037 DEBUG awx.main.dispatch delivered 92294876-ac64-4731-b918-68c451c0afc1 to worker[231] qsize 0
2021-04-24 11:15:34,039 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(91aaff33-c9ca-4caf-a363-ed904c4f16e0, queue=dev-awx-0)
[2021-04-24 11:15:34,039: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(91aaff33-c9ca-4caf-a363-ed904c4f16e0, queue=dev-awx-0)
2021-04-24 11:15:34,050 DEBUG awx.main.dispatch task 92294876-ac64-4731-b918-68c451c0afc1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:15:34,065 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:15:34,065 DEBUG awx.main.dispatch delivered 91aaff33-c9ca-4caf-a363-ed904c4f16e0 to worker[219] qsize 0
2021-04-24 11:15:34,068 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:15:34,074 DEBUG awx.main.dispatch task 92294876-ac64-4731-b918-68c451c0afc1 is finished
2021-04-24 11:15:34,072 DEBUG awx.main.dispatch task 91aaff33-c9ca-4caf-a363-ed904c4f16e0 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:15:34,081 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6fcf1ad2-a5de-43a1-b183-e5069112951a, queue=awx_private_queue)
[2021-04-24 11:15:34,081: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6fcf1ad2-a5de-43a1-b183-e5069112951a, queue=awx_private_queue)
2021-04-24 11:15:34,090 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140190265284160_1" does not exist
2021-04-24 11:15:34,098 DEBUG awx.main.dispatch task 91aaff33-c9ca-4caf-a363-ed904c4f16e0 is finished
2021-04-24 11:15:34,104 DEBUG awx.main.dispatch delivered 6fcf1ad2-a5de-43a1-b183-e5069112951a to worker[219] qsize 0
2021-04-24 11:15:34,109 DEBUG awx.main.dispatch task 6fcf1ad2-a5de-43a1-b183-e5069112951a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:15:34,120 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:15:34,122 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:15:34,127 DEBUG awx.main.dispatch task 6fcf1ad2-a5de-43a1-b183-e5069112951a is finished
2021-04-24 11:15:54,092 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e00a6855-613b-4981-8f39-ca28cc9374c7, queue=awx_private_queue)
[2021-04-24 11:15:54,092: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e00a6855-613b-4981-8f39-ca28cc9374c7, queue=awx_private_queue)
2021-04-24 11:15:54,121 DEBUG awx.main.dispatch delivered e00a6855-613b-4981-8f39-ca28cc9374c7 to worker[231] qsize 0
2021-04-24 11:15:54,130 DEBUG awx.main.dispatch task e00a6855-613b-4981-8f39-ca28cc9374c7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:15:54,146 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:15:54,147 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:15:54,152 DEBUG awx.main.dispatch task e00a6855-613b-4981-8f39-ca28cc9374c7 is finished
RESULT 2
OKREADY
2021-04-24 11:16:04,009 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9bebe608-6b20-4045-b06d-312b37427eee, queue=awx_private_queue)
[2021-04-24 11:16:04,009: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9bebe608-6b20-4045-b06d-312b37427eee, queue=awx_private_queue)
2021-04-24 11:16:04,039 DEBUG awx.main.dispatch delivered 9bebe608-6b20-4045-b06d-312b37427eee to worker[227] qsize 0
2021-04-24 11:16:04,047 DEBUG awx.main.dispatch task 9bebe608-6b20-4045-b06d-312b37427eee starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:16:04,069 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:16:04,071 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:16:04,077 DEBUG awx.main.dispatch task 9bebe608-6b20-4045-b06d-312b37427eee is finished
2021-04-24 11:16:14,038 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(dbcac4ae-4963-439c-bd48-4dd1d3b40bf8, queue=dev-awx-0)
[2021-04-24 11:16:14,038: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(dbcac4ae-4963-439c-bd48-4dd1d3b40bf8, queue=dev-awx-0)
2021-04-24 11:16:14,082 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-24 11:16:14,091 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(908aa624-cdbb-430b-b3a6-5bd9367b0ad6, queue=awx_private_queue)
[2021-04-24 11:16:14,091: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(908aa624-cdbb-430b-b3a6-5bd9367b0ad6, queue=awx_private_queue)
2021-04-24 11:16:14,114 DEBUG awx.main.dispatch delivered 908aa624-cdbb-430b-b3a6-5bd9367b0ad6 to worker[219] qsize 0
2021-04-24 11:16:14,120 DEBUG awx.main.dispatch task 908aa624-cdbb-430b-b3a6-5bd9367b0ad6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:16:14,132 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:16:14,133 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:16:14,139 DEBUG awx.main.dispatch task 908aa624-cdbb-430b-b3a6-5bd9367b0ad6 is finished
2021-04-24 11:16:34,017 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(60d3bdf9-a937-4dae-9ad0-00904339f2cc, queue=awx_private_queue)
[2021-04-24 11:16:34,017: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(60d3bdf9-a937-4dae-9ad0-00904339f2cc, queue=awx_private_queue)
2021-04-24 11:16:34,051 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(60a63b68-29e3-41ac-a095-909d01164e1f, queue=dev-awx-0)
[2021-04-24 11:16:34,051: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(60a63b68-29e3-41ac-a095-909d01164e1f, queue=dev-awx-0)
2021-04-24 11:16:34,050 DEBUG awx.main.dispatch delivered 60d3bdf9-a937-4dae-9ad0-00904339f2cc to worker[219] qsize 0
2021-04-24 11:16:34,074 DEBUG awx.main.dispatch delivered 60a63b68-29e3-41ac-a095-909d01164e1f to worker[225] qsize 0
2021-04-24 11:16:34,065 DEBUG awx.main.dispatch task 60d3bdf9-a937-4dae-9ad0-00904339f2cc starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:16:34,082 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:16:34,086 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:16:34,090 DEBUG awx.main.dispatch task 60d3bdf9-a937-4dae-9ad0-00904339f2cc is finished
2021-04-24 11:16:34,091 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(95366294-87cc-409c-9641-b13fae5df8da, queue=awx_private_queue)
[2021-04-24 11:16:34,091: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(95366294-87cc-409c-9641-b13fae5df8da, queue=awx_private_queue)
2021-04-24 11:16:34,086 DEBUG awx.main.dispatch task 60a63b68-29e3-41ac-a095-909d01164e1f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:16:34,112 DEBUG awx.main.dispatch delivered 95366294-87cc-409c-9641-b13fae5df8da to worker[231] qsize 0
2021-04-24 11:16:34,120 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140190265284160_3" does not exist
2021-04-24 11:16:34,127 DEBUG awx.main.dispatch task 60a63b68-29e3-41ac-a095-909d01164e1f is finished
2021-04-24 11:16:34,121 DEBUG awx.main.dispatch task 95366294-87cc-409c-9641-b13fae5df8da starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:16:34,136 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:16:34,137 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:16:34,141 DEBUG awx.main.dispatch task 95366294-87cc-409c-9641-b13fae5df8da is finished
2021-04-24 11:16:54,094 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c2f5323e-70dc-4361-8fa5-2a3acb72155f, queue=awx_private_queue)
[2021-04-24 11:16:54,094: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c2f5323e-70dc-4361-8fa5-2a3acb72155f, queue=awx_private_queue)
2021-04-24 11:16:54,128 DEBUG awx.main.dispatch delivered c2f5323e-70dc-4361-8fa5-2a3acb72155f to worker[225] qsize 0
2021-04-24 11:16:54,135 DEBUG awx.main.dispatch task c2f5323e-70dc-4361-8fa5-2a3acb72155f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:16:54,148 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:16:54,149 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:16:54,154 DEBUG awx.main.dispatch task c2f5323e-70dc-4361-8fa5-2a3acb72155f is finished
RESULT 2
OKREADY
2021-04-24 11:17:04,019 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c6cedbd2-0aa3-4323-861d-2633d4fc5a0f, queue=awx_private_queue)
[2021-04-24 11:17:04,019: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c6cedbd2-0aa3-4323-861d-2633d4fc5a0f, queue=awx_private_queue)
2021-04-24 11:17:04,066 DEBUG awx.main.dispatch delivered c6cedbd2-0aa3-4323-861d-2633d4fc5a0f to worker[227] qsize 0
2021-04-24 11:17:04,073 DEBUG awx.main.dispatch task c6cedbd2-0aa3-4323-861d-2633d4fc5a0f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:17:04,095 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:17:04,097 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:17:04,102 DEBUG awx.main.dispatch task c6cedbd2-0aa3-4323-861d-2633d4fc5a0f is finished
2021-04-24 11:17:14,033 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d050076d-f22a-4095-853e-ef37585de079, queue=dev-awx-0)
[2021-04-24 11:17:14,033: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d050076d-f22a-4095-853e-ef37585de079, queue=dev-awx-0)
2021-04-24 11:17:14,065 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-24 11:17:14,091 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(67b87399-bfec-4289-9db9-756a2bf2cd71, queue=awx_private_queue)
[2021-04-24 11:17:14,091: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(67b87399-bfec-4289-9db9-756a2bf2cd71, queue=awx_private_queue)
2021-04-24 11:17:14,113 DEBUG awx.main.dispatch delivered 67b87399-bfec-4289-9db9-756a2bf2cd71 to worker[225] qsize 0
2021-04-24 11:17:14,121 DEBUG awx.main.dispatch task 67b87399-bfec-4289-9db9-756a2bf2cd71 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:17:14,134 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:17:14,135 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:17:14,139 DEBUG awx.main.dispatch task 67b87399-bfec-4289-9db9-756a2bf2cd71 is finished
2021-04-24 11:17:34,027 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4a539385-aed4-4c74-9e83-b4bc697bb82c, queue=awx_private_queue)
[2021-04-24 11:17:34,027: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4a539385-aed4-4c74-9e83-b4bc697bb82c, queue=awx_private_queue)
2021-04-24 11:17:34,057 DEBUG awx.main.dispatch delivered 4a539385-aed4-4c74-9e83-b4bc697bb82c to worker[231] qsize 0
2021-04-24 11:17:34,058 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(7c9ed987-2cac-4ecb-912e-46e3efa5f5d8, queue=dev-awx-0)
[2021-04-24 11:17:34,058: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(7c9ed987-2cac-4ecb-912e-46e3efa5f5d8, queue=dev-awx-0)
2021-04-24 11:17:34,069 DEBUG awx.main.dispatch task 4a539385-aed4-4c74-9e83-b4bc697bb82c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:17:34,084 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:17:34,087 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:17:34,087 DEBUG awx.main.dispatch delivered 7c9ed987-2cac-4ecb-912e-46e3efa5f5d8 to worker[225] qsize 0
2021-04-24 11:17:34,091 DEBUG awx.main.dispatch task 4a539385-aed4-4c74-9e83-b4bc697bb82c is finished
2021-04-24 11:17:34,092 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(08601cef-4b1b-4ddd-b0f5-05d86089f357, queue=awx_private_queue)
[2021-04-24 11:17:34,092: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(08601cef-4b1b-4ddd-b0f5-05d86089f357, queue=awx_private_queue)
2021-04-24 11:17:34,095 DEBUG awx.main.dispatch task 7c9ed987-2cac-4ecb-912e-46e3efa5f5d8 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:17:34,113 DEBUG awx.main.dispatch delivered 08601cef-4b1b-4ddd-b0f5-05d86089f357 to worker[219] qsize 0
2021-04-24 11:17:34,117 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140190265284160_4" does not exist
2021-04-24 11:17:34,124 DEBUG awx.main.dispatch task 7c9ed987-2cac-4ecb-912e-46e3efa5f5d8 is finished
2021-04-24 11:17:34,122 DEBUG awx.main.dispatch task 08601cef-4b1b-4ddd-b0f5-05d86089f357 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:17:34,136 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:17:34,137 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:17:34,141 DEBUG awx.main.dispatch task 08601cef-4b1b-4ddd-b0f5-05d86089f357 is finished
2021-04-24 11:17:54,113 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d80e900e-73a3-4ce2-b056-b27406fa2636, queue=awx_private_queue)
[2021-04-24 11:17:54,113: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d80e900e-73a3-4ce2-b056-b27406fa2636, queue=awx_private_queue)
2021-04-24 11:17:54,153 DEBUG awx.main.dispatch delivered d80e900e-73a3-4ce2-b056-b27406fa2636 to worker[219] qsize 0
2021-04-24 11:17:54,168 DEBUG awx.main.dispatch task d80e900e-73a3-4ce2-b056-b27406fa2636 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:17:54,183 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:17:54,185 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:17:54,192 DEBUG awx.main.dispatch task d80e900e-73a3-4ce2-b056-b27406fa2636 is finished
RESULT 2
OKREADY
2021-04-24 11:18:04,027 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(618b3360-1ca2-445f-a3e3-5ac35fa2e417, queue=awx_private_queue)
[2021-04-24 11:18:04,027: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(618b3360-1ca2-445f-a3e3-5ac35fa2e417, queue=awx_private_queue)
2021-04-24 11:18:04,053 DEBUG awx.main.dispatch delivered 618b3360-1ca2-445f-a3e3-5ac35fa2e417 to worker[231] qsize 0
2021-04-24 11:18:04,062 DEBUG awx.main.dispatch task 618b3360-1ca2-445f-a3e3-5ac35fa2e417 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:18:04,076 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:18:04,079 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:18:04,083 DEBUG awx.main.dispatch task 618b3360-1ca2-445f-a3e3-5ac35fa2e417 is finished
2021-04-24 11:18:14,034 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b572c717-5f96-4cee-befb-9adcb7ffd937, queue=dev-awx-0)
[2021-04-24 11:18:14,034: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b572c717-5f96-4cee-befb-9adcb7ffd937, queue=dev-awx-0)
2021-04-24 11:18:14,064 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-24 11:18:14,105 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8241dccd-c651-4a23-a753-4196944dc22e, queue=awx_private_queue)
[2021-04-24 11:18:14,105: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8241dccd-c651-4a23-a753-4196944dc22e, queue=awx_private_queue)
2021-04-24 11:18:14,125 DEBUG awx.main.dispatch delivered 8241dccd-c651-4a23-a753-4196944dc22e to worker[227] qsize 0
2021-04-24 11:18:14,133 DEBUG awx.main.dispatch task 8241dccd-c651-4a23-a753-4196944dc22e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:18:14,150 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:18:14,152 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:18:14,156 DEBUG awx.main.dispatch task 8241dccd-c651-4a23-a753-4196944dc22e is finished
2021-04-24 11:18:34,040 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(fdf630fb-c17e-4630-bc37-10b3f5a87cf4, queue=awx_private_queue)
[2021-04-24 11:18:34,040: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(fdf630fb-c17e-4630-bc37-10b3f5a87cf4, queue=awx_private_queue)
2021-04-24 11:18:34,064 DEBUG awx.main.dispatch delivered fdf630fb-c17e-4630-bc37-10b3f5a87cf4 to worker[225] qsize 0
2021-04-24 11:18:34,065 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(14a0e94a-e7eb-4ce9-921e-581f64d736ab, queue=dev-awx-0)
[2021-04-24 11:18:34,065: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(14a0e94a-e7eb-4ce9-921e-581f64d736ab, queue=dev-awx-0)
2021-04-24 11:18:34,078 DEBUG awx.main.dispatch task fdf630fb-c17e-4630-bc37-10b3f5a87cf4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:18:34,095 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:18:34,091 DEBUG awx.main.dispatch delivered 14a0e94a-e7eb-4ce9-921e-581f64d736ab to worker[231] qsize 0
2021-04-24 11:18:34,097 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:18:34,105 DEBUG awx.main.dispatch task fdf630fb-c17e-4630-bc37-10b3f5a87cf4 is finished
2021-04-24 11:18:34,098 DEBUG awx.main.dispatch task 14a0e94a-e7eb-4ce9-921e-581f64d736ab starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:18:34,106 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(116d2414-04f8-4eb3-90e8-f2c25b9cc439, queue=awx_private_queue)
[2021-04-24 11:18:34,106: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(116d2414-04f8-4eb3-90e8-f2c25b9cc439, queue=awx_private_queue)
2021-04-24 11:18:34,119 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140190265284160_1" does not exist
2021-04-24 11:18:34,128 DEBUG awx.main.dispatch task 14a0e94a-e7eb-4ce9-921e-581f64d736ab is finished
2021-04-24 11:18:34,135 DEBUG awx.main.dispatch delivered 116d2414-04f8-4eb3-90e8-f2c25b9cc439 to worker[231] qsize 0
2021-04-24 11:18:34,142 DEBUG awx.main.dispatch task 116d2414-04f8-4eb3-90e8-f2c25b9cc439 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:18:34,155 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:18:34,157 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:18:34,163 DEBUG awx.main.dispatch task 116d2414-04f8-4eb3-90e8-f2c25b9cc439 is finished
2021-04-24 11:18:54,110 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0b1f9149-3dcd-4c07-99e6-ff01f4eeae31, queue=awx_private_queue)
[2021-04-24 11:18:54,110: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0b1f9149-3dcd-4c07-99e6-ff01f4eeae31, queue=awx_private_queue)
2021-04-24 11:18:54,148 DEBUG awx.main.dispatch delivered 0b1f9149-3dcd-4c07-99e6-ff01f4eeae31 to worker[219] qsize 0
2021-04-24 11:18:54,158 DEBUG awx.main.dispatch task 0b1f9149-3dcd-4c07-99e6-ff01f4eeae31 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:18:54,181 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:18:54,184 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:18:54,191 DEBUG awx.main.dispatch task 0b1f9149-3dcd-4c07-99e6-ff01f4eeae31 is finished
RESULT 2
OKREADY
2021-04-24 11:19:04,042 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6102362b-05a9-4e24-a5e5-008320f812d2, queue=awx_private_queue)
[2021-04-24 11:19:04,042: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6102362b-05a9-4e24-a5e5-008320f812d2, queue=awx_private_queue)
2021-04-24 11:19:04,069 DEBUG awx.main.dispatch delivered 6102362b-05a9-4e24-a5e5-008320f812d2 to worker[219] qsize 0
2021-04-24 11:19:04,078 DEBUG awx.main.dispatch task 6102362b-05a9-4e24-a5e5-008320f812d2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:19:04,093 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:19:04,096 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:19:04,101 DEBUG awx.main.dispatch task 6102362b-05a9-4e24-a5e5-008320f812d2 is finished
2021-04-24 11:19:14,035 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(84767628-be59-4c12-8771-8369b9844408, queue=dev-awx-0)
[2021-04-24 11:19:14,035: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(84767628-be59-4c12-8771-8369b9844408, queue=dev-awx-0)
2021-04-24 11:19:14,075 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-24 11:19:14,106 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8806d005-8d69-4b79-8f8e-2235d56091dc, queue=awx_private_queue)
[2021-04-24 11:19:14,106: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8806d005-8d69-4b79-8f8e-2235d56091dc, queue=awx_private_queue)
2021-04-24 11:19:14,128 DEBUG awx.main.dispatch delivered 8806d005-8d69-4b79-8f8e-2235d56091dc to worker[225] qsize 0
2021-04-24 11:19:14,134 DEBUG awx.main.dispatch task 8806d005-8d69-4b79-8f8e-2235d56091dc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:19:14,146 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:19:14,148 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:19:14,151 DEBUG awx.main.dispatch task 8806d005-8d69-4b79-8f8e-2235d56091dc is finished
2021-04-24 11:19:34,050 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(eff49832-cacc-46d9-a0d0-273f773452a8, queue=awx_private_queue)
[2021-04-24 11:19:34,050: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(eff49832-cacc-46d9-a0d0-273f773452a8, queue=awx_private_queue)
2021-04-24 11:19:34,085 DEBUG awx.main.dispatch delivered eff49832-cacc-46d9-a0d0-273f773452a8 to worker[219] qsize 0
2021-04-24 11:19:34,087 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(512fa4e2-8de2-44f9-8e33-48460d726fb0, queue=dev-awx-0)
[2021-04-24 11:19:34,087: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(512fa4e2-8de2-44f9-8e33-48460d726fb0, queue=dev-awx-0)
2021-04-24 11:19:34,097 DEBUG awx.main.dispatch task eff49832-cacc-46d9-a0d0-273f773452a8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:19:34,118 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:19:34,117 DEBUG awx.main.dispatch delivered 512fa4e2-8de2-44f9-8e33-48460d726fb0 to worker[227] qsize 0
2021-04-24 11:19:34,120 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:19:34,119 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(377b2ddf-ef53-402b-b3a0-47e3ee583b4e, queue=awx_private_queue)
[2021-04-24 11:19:34,119: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(377b2ddf-ef53-402b-b3a0-47e3ee583b4e, queue=awx_private_queue)
2021-04-24 11:19:34,125 DEBUG awx.main.dispatch task eff49832-cacc-46d9-a0d0-273f773452a8 is finished
2021-04-24 11:19:34,140 DEBUG awx.main.dispatch delivered 377b2ddf-ef53-402b-b3a0-47e3ee583b4e to worker[225] qsize 0
2021-04-24 11:19:34,125 DEBUG awx.main.dispatch task 512fa4e2-8de2-44f9-8e33-48460d726fb0 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:19:34,148 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140190265284160_2" does not exist
2021-04-24 11:19:34,153 DEBUG awx.main.dispatch task 512fa4e2-8de2-44f9-8e33-48460d726fb0 is finished
2021-04-24 11:19:34,146 DEBUG awx.main.dispatch task 377b2ddf-ef53-402b-b3a0-47e3ee583b4e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:19:34,162 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:19:34,164 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:19:34,167 DEBUG awx.main.dispatch task 377b2ddf-ef53-402b-b3a0-47e3ee583b4e is finished
2021-04-24 11:19:54,129 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4aeb1ca9-3cf8-4ec7-a999-41759bb48206, queue=awx_private_queue)
[2021-04-24 11:19:54,129: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4aeb1ca9-3cf8-4ec7-a999-41759bb48206, queue=awx_private_queue)
2021-04-24 11:19:54,152 DEBUG awx.main.dispatch delivered 4aeb1ca9-3cf8-4ec7-a999-41759bb48206 to worker[225] qsize 0
2021-04-24 11:19:54,164 DEBUG awx.main.dispatch task 4aeb1ca9-3cf8-4ec7-a999-41759bb48206 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:19:54,181 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:19:54,183 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:19:54,189 DEBUG awx.main.dispatch task 4aeb1ca9-3cf8-4ec7-a999-41759bb48206 is finished
RESULT 2
OKREADY
2021-04-24 11:20:04,051 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7d6a5a59-b648-42aa-bdc3-c0570d314dad, queue=awx_private_queue)
[2021-04-24 11:20:04,051: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7d6a5a59-b648-42aa-bdc3-c0570d314dad, queue=awx_private_queue)
2021-04-24 11:20:04,078 DEBUG awx.main.dispatch delivered 7d6a5a59-b648-42aa-bdc3-c0570d314dad to worker[219] qsize 0
2021-04-24 11:20:04,084 DEBUG awx.main.dispatch task 7d6a5a59-b648-42aa-bdc3-c0570d314dad starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:20:04,097 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:20:04,099 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:20:04,103 DEBUG awx.main.dispatch task 7d6a5a59-b648-42aa-bdc3-c0570d314dad is finished
2021-04-24 11:20:14,035 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(bf06c288-087e-4b2e-b932-ace309f364c7, queue=dev-awx-0)
[2021-04-24 11:20:14,035: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(bf06c288-087e-4b2e-b932-ace309f364c7, queue=dev-awx-0)
2021-04-24 11:20:14,073 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-24 11:20:14,129 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(da2b5b52-0cb0-47c1-9d5a-462a8f6524f1, queue=awx_private_queue)
[2021-04-24 11:20:14,129: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(da2b5b52-0cb0-47c1-9d5a-462a8f6524f1, queue=awx_private_queue)
2021-04-24 11:20:14,153 DEBUG awx.main.dispatch delivered da2b5b52-0cb0-47c1-9d5a-462a8f6524f1 to worker[231] qsize 0
2021-04-24 11:20:14,164 DEBUG awx.main.dispatch task da2b5b52-0cb0-47c1-9d5a-462a8f6524f1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:20:14,192 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:20:14,195 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:20:14,203 DEBUG awx.main.dispatch task da2b5b52-0cb0-47c1-9d5a-462a8f6524f1 is finished
2021-04-24 11:20:34,066 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(871bc74f-612c-4bf3-a8d6-cf234f5e112c, queue=awx_private_queue)
[2021-04-24 11:20:34,066: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(871bc74f-612c-4bf3-a8d6-cf234f5e112c, queue=awx_private_queue)
2021-04-24 11:20:34,111 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(49f5cd1f-cbcc-4b0c-94d1-d2bac72f6315, queue=dev-awx-0)
[2021-04-24 11:20:34,111: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(49f5cd1f-cbcc-4b0c-94d1-d2bac72f6315, queue=dev-awx-0)
2021-04-24 11:20:34,112 DEBUG awx.main.dispatch delivered 871bc74f-612c-4bf3-a8d6-cf234f5e112c to worker[227] qsize 0
2021-04-24 11:20:34,128 DEBUG awx.main.dispatch task 871bc74f-612c-4bf3-a8d6-cf234f5e112c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:20:34,143 DEBUG awx.main.dispatch delivered 49f5cd1f-cbcc-4b0c-94d1-d2bac72f6315 to worker[231] qsize 0
2021-04-24 11:20:34,150 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:20:34,144 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1d3888dd-731c-4e70-97ef-ce99a3d0bed1, queue=awx_private_queue)
[2021-04-24 11:20:34,144: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1d3888dd-731c-4e70-97ef-ce99a3d0bed1, queue=awx_private_queue)
2021-04-24 11:20:34,161 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:20:34,165 DEBUG awx.main.dispatch delivered 1d3888dd-731c-4e70-97ef-ce99a3d0bed1 to worker[219] qsize 0
2021-04-24 11:20:34,167 DEBUG awx.main.dispatch task 871bc74f-612c-4bf3-a8d6-cf234f5e112c is finished
2021-04-24 11:20:34,152 DEBUG awx.main.dispatch task 49f5cd1f-cbcc-4b0c-94d1-d2bac72f6315 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:20:34,174 DEBUG awx.main.dispatch task 1d3888dd-731c-4e70-97ef-ce99a3d0bed1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:20:34,188 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:20:34,184 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140190265284160_2" does not exist
2021-04-24 11:20:34,190 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:20:34,193 DEBUG awx.main.dispatch task 49f5cd1f-cbcc-4b0c-94d1-d2bac72f6315 is finished
2021-04-24 11:20:34,194 DEBUG awx.main.dispatch task 1d3888dd-731c-4e70-97ef-ce99a3d0bed1 is finished
2021-04-24 11:20:54,152 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(44f9f5c1-325e-4dfa-82aa-1630d8a84dbe, queue=awx_private_queue)
[2021-04-24 11:20:54,152: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(44f9f5c1-325e-4dfa-82aa-1630d8a84dbe, queue=awx_private_queue)
2021-04-24 11:20:54,179 DEBUG awx.main.dispatch delivered 44f9f5c1-325e-4dfa-82aa-1630d8a84dbe to worker[219] qsize 0
2021-04-24 11:20:54,188 DEBUG awx.main.dispatch task 44f9f5c1-325e-4dfa-82aa-1630d8a84dbe starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:20:54,199 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:20:54,201 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:20:54,205 DEBUG awx.main.dispatch task 44f9f5c1-325e-4dfa-82aa-1630d8a84dbe is finished
RESULT 2
OKREADY
2021-04-24 11:21:04,067 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8a0e8e87-f4ae-4ed3-a860-c08b898c8496, queue=awx_private_queue)
[2021-04-24 11:21:04,067: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8a0e8e87-f4ae-4ed3-a860-c08b898c8496, queue=awx_private_queue)
2021-04-24 11:21:04,116 DEBUG awx.main.dispatch delivered 8a0e8e87-f4ae-4ed3-a860-c08b898c8496 to worker[227] qsize 0
2021-04-24 11:21:04,131 DEBUG awx.main.dispatch task 8a0e8e87-f4ae-4ed3-a860-c08b898c8496 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:21:04,171 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:21:04,174 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:21:04,179 DEBUG awx.main.dispatch task 8a0e8e87-f4ae-4ed3-a860-c08b898c8496 is finished
2021-04-24 11:21:14,036 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e995d14f-656f-4aed-a400-9e5ab6dc0b8d, queue=dev-awx-0)
[2021-04-24 11:21:14,036: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e995d14f-656f-4aed-a400-9e5ab6dc0b8d, queue=dev-awx-0)
2021-04-24 11:21:14,077 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-24 11:21:14,151 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5dfdb0dd-9627-4598-bda8-540a1a0ab5e8, queue=awx_private_queue)
[2021-04-24 11:21:14,151: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5dfdb0dd-9627-4598-bda8-540a1a0ab5e8, queue=awx_private_queue)
2021-04-24 11:21:14,193 DEBUG awx.main.dispatch delivered 5dfdb0dd-9627-4598-bda8-540a1a0ab5e8 to worker[219] qsize 0
2021-04-24 11:21:14,201 DEBUG awx.main.dispatch task 5dfdb0dd-9627-4598-bda8-540a1a0ab5e8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:21:14,216 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:21:14,218 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:21:14,222 DEBUG awx.main.dispatch task 5dfdb0dd-9627-4598-bda8-540a1a0ab5e8 is finished
2021-04-24 11:21:34,072 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f87c15ad-f37f-4c88-8f30-ac8a773dcb71, queue=awx_private_queue)
[2021-04-24 11:21:34,072: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f87c15ad-f37f-4c88-8f30-ac8a773dcb71, queue=awx_private_queue)
2021-04-24 11:21:34,117 DEBUG awx.main.dispatch delivered f87c15ad-f37f-4c88-8f30-ac8a773dcb71 to worker[219] qsize 0
2021-04-24 11:21:34,120 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(294b8b66-90e0-4de9-904f-9126dd7fc74d, queue=dev-awx-0)
[2021-04-24 11:21:34,120: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(294b8b66-90e0-4de9-904f-9126dd7fc74d, queue=dev-awx-0)
2021-04-24 11:21:34,132 DEBUG awx.main.dispatch task f87c15ad-f37f-4c88-8f30-ac8a773dcb71 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:21:34,149 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:21:34,151 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:21:34,150 DEBUG awx.main.dispatch delivered 294b8b66-90e0-4de9-904f-9126dd7fc74d to worker[225] qsize 0
2021-04-24 11:21:34,156 DEBUG awx.main.dispatch task f87c15ad-f37f-4c88-8f30-ac8a773dcb71 is finished
2021-04-24 11:21:34,152 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c81b1d53-e403-429c-8fd5-84c8c67af804, queue=awx_private_queue)
[2021-04-24 11:21:34,152: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c81b1d53-e403-429c-8fd5-84c8c67af804, queue=awx_private_queue)
2021-04-24 11:21:34,174 DEBUG awx.main.dispatch delivered c81b1d53-e403-429c-8fd5-84c8c67af804 to worker[227] qsize 0
2021-04-24 11:21:34,161 DEBUG awx.main.dispatch task 294b8b66-90e0-4de9-904f-9126dd7fc74d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:21:34,184 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140190265284160_5" does not exist
2021-04-24 11:21:34,189 DEBUG awx.main.dispatch task 294b8b66-90e0-4de9-904f-9126dd7fc74d is finished
2021-04-24 11:21:34,182 DEBUG awx.main.dispatch task c81b1d53-e403-429c-8fd5-84c8c67af804 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:21:34,199 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:21:34,201 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:21:34,205 DEBUG awx.main.dispatch task c81b1d53-e403-429c-8fd5-84c8c67af804 is finished
2021-04-24 11:21:54,165 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c2cb5ed2-f977-47a9-9299-0da932781c20, queue=awx_private_queue)
[2021-04-24 11:21:54,165: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c2cb5ed2-f977-47a9-9299-0da932781c20, queue=awx_private_queue)
2021-04-24 11:21:54,189 DEBUG awx.main.dispatch delivered c2cb5ed2-f977-47a9-9299-0da932781c20 to worker[219] qsize 0
2021-04-24 11:21:54,202 DEBUG awx.main.dispatch task c2cb5ed2-f977-47a9-9299-0da932781c20 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:21:54,215 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:21:54,217 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:21:54,222 DEBUG awx.main.dispatch task c2cb5ed2-f977-47a9-9299-0da932781c20 is finished
RESULT 2
OKREADY
2021-04-24 11:22:04,071 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(36bf241e-c429-4f97-820c-2c70c732e634, queue=awx_private_queue)
[2021-04-24 11:22:04,071: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(36bf241e-c429-4f97-820c-2c70c732e634, queue=awx_private_queue)
2021-04-24 11:22:04,098 DEBUG awx.main.dispatch delivered 36bf241e-c429-4f97-820c-2c70c732e634 to worker[219] qsize 0
2021-04-24 11:22:04,107 DEBUG awx.main.dispatch task 36bf241e-c429-4f97-820c-2c70c732e634 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:22:04,118 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:22:04,120 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:22:04,124 DEBUG awx.main.dispatch task 36bf241e-c429-4f97-820c-2c70c732e634 is finished
2021-04-24 11:22:14,035 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(64c0257a-0950-4487-8e18-9ce6cc1b9665, queue=dev-awx-0)
[2021-04-24 11:22:14,035: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(64c0257a-0950-4487-8e18-9ce6cc1b9665, queue=dev-awx-0)
2021-04-24 11:22:14,076 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-24 11:22:14,164 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c0f9c224-35fe-420e-ab30-55da8be1abc7, queue=awx_private_queue)
[2021-04-24 11:22:14,164: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c0f9c224-35fe-420e-ab30-55da8be1abc7, queue=awx_private_queue)
2021-04-24 11:22:14,195 DEBUG awx.main.dispatch delivered c0f9c224-35fe-420e-ab30-55da8be1abc7 to worker[231] qsize 0
2021-04-24 11:22:14,203 DEBUG awx.main.dispatch task c0f9c224-35fe-420e-ab30-55da8be1abc7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:22:14,223 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:22:14,225 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:22:14,229 DEBUG awx.main.dispatch task c0f9c224-35fe-420e-ab30-55da8be1abc7 is finished
2021-04-24 11:22:34,089 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bf3f0b0c-becc-41a0-9500-a93ad3626536, queue=awx_private_queue)
[2021-04-24 11:22:34,089: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bf3f0b0c-becc-41a0-9500-a93ad3626536, queue=awx_private_queue)
2021-04-24 11:22:34,135 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(001b7e9f-226a-488c-805a-f62a04e43106, queue=dev-awx-0)
[2021-04-24 11:22:34,135: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(001b7e9f-226a-488c-805a-f62a04e43106, queue=dev-awx-0)
2021-04-24 11:22:34,135 DEBUG awx.main.dispatch delivered bf3f0b0c-becc-41a0-9500-a93ad3626536 to worker[219] qsize 0
2021-04-24 11:22:34,150 DEBUG awx.main.dispatch task bf3f0b0c-becc-41a0-9500-a93ad3626536 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:22:34,167 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:22:34,165 DEBUG awx.main.dispatch delivered 001b7e9f-226a-488c-805a-f62a04e43106 to worker[231] qsize 0
2021-04-24 11:22:34,170 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:22:34,174 DEBUG awx.main.dispatch task bf3f0b0c-becc-41a0-9500-a93ad3626536 is finished
2021-04-24 11:22:34,168 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7f598fd4-8841-468b-995b-95076404a17e, queue=awx_private_queue)
[2021-04-24 11:22:34,168: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7f598fd4-8841-468b-995b-95076404a17e, queue=awx_private_queue)
2021-04-24 11:22:34,172 DEBUG awx.main.dispatch task 001b7e9f-226a-488c-805a-f62a04e43106 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:22:34,190 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140190265284160_3" does not exist
2021-04-24 11:22:34,196 DEBUG awx.main.dispatch task 001b7e9f-226a-488c-805a-f62a04e43106 is finished
2021-04-24 11:22:34,197 DEBUG awx.main.dispatch delivered 7f598fd4-8841-468b-995b-95076404a17e to worker[225] qsize 0
2021-04-24 11:22:34,205 DEBUG awx.main.dispatch task 7f598fd4-8841-468b-995b-95076404a17e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:22:34,222 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:22:34,224 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:22:34,228 DEBUG awx.main.dispatch task 7f598fd4-8841-468b-995b-95076404a17e is finished
2021-04-24 11:22:54,179 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(49f502f8-6d5c-4d44-b1bd-ab1ea81241db, queue=awx_private_queue)
[2021-04-24 11:22:54,179: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(49f502f8-6d5c-4d44-b1bd-ab1ea81241db, queue=awx_private_queue)
2021-04-24 11:22:54,204 DEBUG awx.main.dispatch delivered 49f502f8-6d5c-4d44-b1bd-ab1ea81241db to worker[219] qsize 0
2021-04-24 11:22:54,212 DEBUG awx.main.dispatch task 49f502f8-6d5c-4d44-b1bd-ab1ea81241db starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:22:54,228 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:22:54,231 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:22:54,240 DEBUG awx.main.dispatch task 49f502f8-6d5c-4d44-b1bd-ab1ea81241db is finished
RESULT 2
OKREADY
2021-04-24 11:23:04,084 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(53f240d6-8b84-427e-874a-7e3923ab6963, queue=awx_private_queue)
[2021-04-24 11:23:04,084: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(53f240d6-8b84-427e-874a-7e3923ab6963, queue=awx_private_queue)
2021-04-24 11:23:04,110 DEBUG awx.main.dispatch delivered 53f240d6-8b84-427e-874a-7e3923ab6963 to worker[227] qsize 0
2021-04-24 11:23:04,123 DEBUG awx.main.dispatch task 53f240d6-8b84-427e-874a-7e3923ab6963 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:23:04,153 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:23:04,155 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:23:04,160 DEBUG awx.main.dispatch task 53f240d6-8b84-427e-874a-7e3923ab6963 is finished
2021-04-24 11:23:14,035 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8c53c2b1-7a54-4300-8b09-4e418dcec881, queue=dev-awx-0)
[2021-04-24 11:23:14,035: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8c53c2b1-7a54-4300-8b09-4e418dcec881, queue=dev-awx-0)
2021-04-24 11:23:14,064 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-24 11:23:14,178 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d8b12e40-2bbc-42d9-b330-49a6d8e1bf02, queue=awx_private_queue)
[2021-04-24 11:23:14,178: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d8b12e40-2bbc-42d9-b330-49a6d8e1bf02, queue=awx_private_queue)
2021-04-24 11:23:14,205 DEBUG awx.main.dispatch delivered d8b12e40-2bbc-42d9-b330-49a6d8e1bf02 to worker[231] qsize 0
2021-04-24 11:23:14,215 DEBUG awx.main.dispatch task d8b12e40-2bbc-42d9-b330-49a6d8e1bf02 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:23:14,234 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:23:14,236 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:23:14,243 DEBUG awx.main.dispatch task d8b12e40-2bbc-42d9-b330-49a6d8e1bf02 is finished
2021-04-24 11:23:34,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(79c88bfd-1621-4f4e-90fb-4daf4681b4b3, queue=awx_private_queue)
[2021-04-24 11:23:34,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(79c88bfd-1621-4f4e-90fb-4daf4681b4b3, queue=awx_private_queue)
2021-04-24 11:23:34,119 DEBUG awx.main.dispatch delivered 79c88bfd-1621-4f4e-90fb-4daf4681b4b3 to worker[227] qsize 0
2021-04-24 11:23:34,125 DEBUG awx.main.dispatch task 79c88bfd-1621-4f4e-90fb-4daf4681b4b3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:23:34,137 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:23:34,139 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:23:34,143 DEBUG awx.main.dispatch task 79c88bfd-1621-4f4e-90fb-4daf4681b4b3 is finished
2021-04-24 11:23:34,135 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(eb323409-0a85-48ed-9f57-b3b7a2aa24a2, queue=dev-awx-0)
[2021-04-24 11:23:34,135: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(eb323409-0a85-48ed-9f57-b3b7a2aa24a2, queue=dev-awx-0)
2021-04-24 11:23:34,174 DEBUG awx.main.dispatch delivered eb323409-0a85-48ed-9f57-b3b7a2aa24a2 to worker[219] qsize 0
2021-04-24 11:23:34,179 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(33dbcb58-9fb0-4deb-a6e7-0acc4a76c51f, queue=awx_private_queue)
[2021-04-24 11:23:34,179: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(33dbcb58-9fb0-4deb-a6e7-0acc4a76c51f, queue=awx_private_queue)
2021-04-24 11:23:34,182 DEBUG awx.main.dispatch task eb323409-0a85-48ed-9f57-b3b7a2aa24a2 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-04-24 11:23:34,202 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140190265284160_2" does not exist
2021-04-24 11:23:34,209 DEBUG awx.main.dispatch task eb323409-0a85-48ed-9f57-b3b7a2aa24a2 is finished
2021-04-24 11:23:34,211 DEBUG awx.main.dispatch delivered 33dbcb58-9fb0-4deb-a6e7-0acc4a76c51f to worker[225] qsize 0
2021-04-24 11:23:34,220 DEBUG awx.main.dispatch task 33dbcb58-9fb0-4deb-a6e7-0acc4a76c51f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:23:34,236 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:23:34,237 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:23:34,241 DEBUG awx.main.dispatch task 33dbcb58-9fb0-4deb-a6e7-0acc4a76c51f is finished
2021-04-24 11:23:54,197 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c9039b2f-6f7a-4dfe-a432-0abf9bfeaaa2, queue=awx_private_queue)
[2021-04-24 11:23:54,197: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c9039b2f-6f7a-4dfe-a432-0abf9bfeaaa2, queue=awx_private_queue)
2021-04-24 11:23:54,234 DEBUG awx.main.dispatch delivered c9039b2f-6f7a-4dfe-a432-0abf9bfeaaa2 to worker[231] qsize 0
2021-04-24 11:23:54,245 DEBUG awx.main.dispatch task c9039b2f-6f7a-4dfe-a432-0abf9bfeaaa2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:23:54,259 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:23:54,261 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:23:54,270 DEBUG awx.main.dispatch task c9039b2f-6f7a-4dfe-a432-0abf9bfeaaa2 is finished
RESULT 2
OKREADY
2021-04-24 11:24:04,092 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4bb8ac12-8542-462b-a94f-f0985302c8a4, queue=awx_private_queue)
[2021-04-24 11:24:04,092: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4bb8ac12-8542-462b-a94f-f0985302c8a4, queue=awx_private_queue)
2021-04-24 11:24:04,118 DEBUG awx.main.dispatch delivered 4bb8ac12-8542-462b-a94f-f0985302c8a4 to worker[219] qsize 0
2021-04-24 11:24:04,129 DEBUG awx.main.dispatch task 4bb8ac12-8542-462b-a94f-f0985302c8a4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:24:04,157 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:24:04,160 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-04-24 11:24:04,167 DEBUG awx.main.dispatch task 4bb8ac12-8542-462b-a94f-f0985302c8a4 is finished
2021-04-24 11:24:14,035 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(ea77b637-f50a-435f-bc09-b42ad20210ef, queue=dev-awx-0)
[2021-04-24 11:24:14,035: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(ea77b637-f50a-435f-bc09-b42ad20210ef, queue=dev-awx-0)
2021-04-24 11:24:14,082 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-04-24 11:24:14,190 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(97e33a22-720b-4e9b-886d-780923fb2187, queue=awx_private_queue)
[2021-04-24 11:24:14,190: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(97e33a22-720b-4e9b-886d-780923fb2187, queue=awx_private_queue)
2021-04-24 11:24:14,213 DEBUG awx.main.dispatch delivered 97e33a22-720b-4e9b-886d-780923fb2187 to worker[225] qsize 0
2021-04-24 11:24:14,220 DEBUG awx.main.dispatch task 97e33a22-720b-4e9b-886d-780923fb2187 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-04-24 11:24:14,238 DEBUG awx.main.scheduler Running Tower task manager.
2021-04-24 11:24:14,240 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-04-24 11:24:14,245 DEBUG awx.main.dispatch task 97e33a22-720b-4e9b-886d-780923fb2187 is finished
2021-04-24 11:24:34,103 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(78b44de9-48c6-4691-881c-a0531880ed3b, queue=awx_private_queue)
[2021-04-24 11:24:34,103: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(78b44de9-48c6-4691-881c-a0531880ed3b, queue=awx_private_queue)
2021-04-24 11:24:34,132 DEBUG awx.main.dispatch delivered 78b44de9-48c6-4691-881c-a0531880ed3b to worker[225] qsize 0
2021-04-24 11:24:34,140 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(e528594c-7f3e-49f2-aa5b-da120c6c0fbd, queue=dev-awx-0)
[2021-04-24 11:24:34,140: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(e528594c-7f3e-49f2-aa5b-da120c6c0fbd, queue=dev-awx-0)
2021-04-24 11:24:34,143 DEBUG awx.main.dispatch task 78b44de9-48c6-4691-881c-a0531880ed3b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-04-24 11:24:34,158 DEBUG awx.main.tasks Starting periodic scheduler
2021-04-24 11:24:34,161 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, con