By type
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"elapsed": 0,
"match_groupdict": {},
"match_groups": [],
"path": null,
"port": 5432,
"search_regex": null,
"state": "started"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"db": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
}
Default organization added.
Demo Credential, Inventory, and Job Template added.
(changed: True)
2021-05-03 22:45:44,938 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(5b3f7393-9e93-491f-ad9b-c56696c7e713, queue=awx_private_queue)
Successfully registered instance dev-awx-0
(changed: True)
2021-05-03 22:45:49,066 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(7daa2fd0-b786-431a-9ea1-6ebf21fcd93d, queue=awx_private_queue)
2021-05-03 22:45:49,104 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(1148f35b-86ef-4a39-a9c3-b921afc47817, queue=awx_private_queue)
Creating instance group tower
(changed: True)
/usr/bin/launch_awx_task.sh: line 27: unset: `#!/bin/sh': not a valid identifier
2021-05-03 22:45:49,956 INFO RPC interface 'supervisor' initialized
2021-05-03 22:45:49,956 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2021-05-03 22:45:49,957 INFO supervisord started with pid 151
2021-05-03 22:45:50,964 INFO spawned: 'awx-config-watcher' with pid 154
2021-05-03 22:45:50,967 INFO spawned: 'dispatcher' with pid 155
2021-05-03 22:45:50,970 INFO spawned: 'callback-receiver' with pid 156
2021-05-03 22:45:50,973 INFO spawned: 'channels-worker' with pid 157
READY
2021-05-03 22:45:52,106 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-03 22:45:52,106 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-03 22:45:52,107 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-03 22:45:52,107 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-03 22:45:54,166 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-05-03 22:45:54,166 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-05-03 22:45:54,174 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2021-05-03 22:45:54,174 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2021-05-03 22:45:54,182 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2021-05-03 22:45:54,182 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2021-05-03 22:45:54,191 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2021-05-03 22:45:54,191 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2021-05-03 22:45:54,333 ERROR celery.beat Removing corrupted schedule file '/var/lib/awx/beat.db': error(11, 'Resource temporarily unavailable')
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 485, in setup_schedule
self._store = self._open_schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 475, in _open_schedule
return self.persistence.open(self.schedule_filename, writeback=True)
File "/usr/lib64/python3.6/shelve.py", line 243, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/usr/lib64/python3.6/shelve.py", line 227, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
File "/usr/lib64/python3.6/dbm/__init__.py", line 94, in open
return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable
2021-05-03 22:45:54,334 WARNING awx.main.dispatch scaling up worker pid:181
2021-05-03 22:45:54,350 WARNING awx.main.dispatch scaling up worker pid:182
2021-05-03 22:45:54,373 - INFO - runworker - Using single-threaded worker.
2021-05-03 22:45:54,365 WARNING awx.main.dispatch scaling up worker pid:183
2021-05-03 22:45:54,373 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer)
2021-05-03 22:45:54,374 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive
2021-05-03 22:45:54,381 WARNING awx.main.dispatch scaling up worker pid:184
2021-05-03 22:45:54,389 DEBUG awx.main.tasks Syncing Schedules
2021-05-03 22:45:54,616 DEBUG awx.main.tasks Waited 0.0009038448333740234 seconds to obtain lock name: cluster_policy_lock
2021-05-03 22:45:54,638 DEBUG awx.main.tasks Total non-isolated instances:1 available for policy: 1
2021-05-03 22:45:54,638 DEBUG awx.main.tasks Policy percentage, adding Instances [1] to Group tower
2021-05-03 22:45:54,639 DEBUG awx.main.tasks Adding instances [1] to group tower
2021-05-03 22:45:54,655 DEBUG awx.main.tasks Cluster policy computation finished in 0.03869485855102539 seconds
2021-05-03 22:45:54,656 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 22:45:54,693 DEBUG awx.main.dispatch listening on ['awx_private_queue [direct]', 'dev-awx-0 [direct]', 'tower_broadcast_all_dev-awx-0 [fanout]']
RESULT 2
OKREADY
2021-05-03 22:46:14,522 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(dae05be3-b284-4c9c-a6cf-23d8a843c838, queue=awx_private_queue)
[2021-05-03 22:46:14,522: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(dae05be3-b284-4c9c-a6cf-23d8a843c838, queue=awx_private_queue)
2021-05-03 22:46:14,557 DEBUG awx.main.dispatch delivered dae05be3-b284-4c9c-a6cf-23d8a843c838 to worker[182] qsize 0
2021-05-03 22:46:14,572 DEBUG awx.main.dispatch task dae05be3-b284-4c9c-a6cf-23d8a843c838 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:46:14,583 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:46:14,619 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:46:14,678 DEBUG awx.main.dispatch task dae05be3-b284-4c9c-a6cf-23d8a843c838 is finished
2021-05-03 22:46:24,475 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(cb80d071-9451-4405-b570-9d887a8be758, queue=awx_private_queue)
[2021-05-03 22:46:24,475: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(cb80d071-9451-4405-b570-9d887a8be758, queue=awx_private_queue)
2021-05-03 22:46:24,492 DEBUG awx.main.dispatch delivered cb80d071-9451-4405-b570-9d887a8be758 to worker[181] qsize 0
2021-05-03 22:46:24,497 DEBUG awx.main.dispatch task cb80d071-9451-4405-b570-9d887a8be758 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:46:24,517 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:46:24,540 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:46:24.533628+00:00
2021-05-03 22:46:24,576 DEBUG awx.main.dispatch task cb80d071-9451-4405-b570-9d887a8be758 is finished
2021-05-03 22:46:34,493 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(47e58179-4931-45cc-94b3-2755f471383e, queue=awx_private_queue)
[2021-05-03 22:46:34,493: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(47e58179-4931-45cc-94b3-2755f471383e, queue=awx_private_queue)
2021-05-03 22:46:34,512 DEBUG awx.main.dispatch delivered 47e58179-4931-45cc-94b3-2755f471383e to worker[184] qsize 0
2021-05-03 22:46:34,519 DEBUG awx.main.dispatch task 47e58179-4931-45cc-94b3-2755f471383e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:46:34,525 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:46:34,566 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:46:34,611 DEBUG awx.main.dispatch task 47e58179-4931-45cc-94b3-2755f471383e is finished
2021-05-03 22:46:54,474 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(661c31c1-86a0-4839-a7c5-d4b359502ef6, queue=dev-awx-0)
[2021-05-03 22:46:54,474: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(661c31c1-86a0-4839-a7c5-d4b359502ef6, queue=dev-awx-0)
2021-05-03 22:46:54,734 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(756b1297-abc5-4f82-ada4-db3379dcb909, queue=dev-awx-0)
[2021-05-03 22:46:54,734: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(756b1297-abc5-4f82-ada4-db3379dcb909, queue=dev-awx-0)
2021-05-03 22:46:54,749 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4cee5dd8-5d4f-4632-8c56-0ab930ab7cc4, queue=awx_private_queue)
[2021-05-03 22:46:54,749: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4cee5dd8-5d4f-4632-8c56-0ab930ab7cc4, queue=awx_private_queue)
2021-05-03 22:46:54,764 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(425e0aa6-8b81-42ca-bca9-df71715553e0, queue=awx_private_queue)
[2021-05-03 22:46:54,764: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(425e0aa6-8b81-42ca-bca9-df71715553e0, queue=awx_private_queue)
2021-05-03 22:46:54,806 DEBUG awx.main.dispatch delivered 661c31c1-86a0-4839-a7c5-d4b359502ef6 to worker[184] qsize 0
2021-05-03 22:46:54,809 DEBUG awx.main.dispatch task 661c31c1-86a0-4839-a7c5-d4b359502ef6 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 22:46:54,812 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 22:46:54,811 DEBUG awx.main.dispatch delivered 756b1297-abc5-4f82-ada4-db3379dcb909 to worker[182] qsize 0
2021-05-03 22:46:54,818 DEBUG awx.main.dispatch task 756b1297-abc5-4f82-ada4-db3379dcb909 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 22:46:54,825 DEBUG awx.main.dispatch delivered 4cee5dd8-5d4f-4632-8c56-0ab930ab7cc4 to worker[181] qsize 0
2021-05-03 22:46:54,827 DEBUG awx.main.dispatch delivered 425e0aa6-8b81-42ca-bca9-df71715553e0 to worker[183] qsize 0
2021-05-03 22:46:54,827 DEBUG awx.main.dispatch task 4cee5dd8-5d4f-4632-8c56-0ab930ab7cc4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:46:54,836 DEBUG awx.main.dispatch task 425e0aa6-8b81-42ca-bca9-df71715553e0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:46:54,848 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:46:54,850 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:46:54,857 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:46:24.517597+00:00
2021-05-03 22:46:54,860 DEBUG awx.main.dispatch task 756b1297-abc5-4f82-ada4-db3379dcb909 is finished
2021-05-03 22:46:54,877 DEBUG awx.main.dispatch task 661c31c1-86a0-4839-a7c5-d4b359502ef6 is finished
2021-05-03 22:46:54,880 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:46:54,895 DEBUG awx.main.dispatch task 4cee5dd8-5d4f-4632-8c56-0ab930ab7cc4 is finished
2021-05-03 22:46:54,964 DEBUG awx.main.dispatch task 425e0aa6-8b81-42ca-bca9-df71715553e0 is finished
RESULT 2
OKREADY
2021-05-03 22:47:14,780 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(33839d01-f54b-4de4-97c7-02d19eb15a1c, queue=awx_private_queue)
[2021-05-03 22:47:14,780: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(33839d01-f54b-4de4-97c7-02d19eb15a1c, queue=awx_private_queue)
2021-05-03 22:47:14,820 DEBUG awx.main.dispatch delivered 33839d01-f54b-4de4-97c7-02d19eb15a1c to worker[182] qsize 0
2021-05-03 22:47:14,826 DEBUG awx.main.dispatch task 33839d01-f54b-4de4-97c7-02d19eb15a1c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:47:14,829 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:47:14,864 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:47:14,912 DEBUG awx.main.dispatch task 33839d01-f54b-4de4-97c7-02d19eb15a1c is finished
2021-05-03 22:47:24,750 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6d9bf0f3-77ee-4a1b-b1bd-795637677ac5, queue=awx_private_queue)
[2021-05-03 22:47:24,750: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6d9bf0f3-77ee-4a1b-b1bd-795637677ac5, queue=awx_private_queue)
2021-05-03 22:47:24,764 DEBUG awx.main.dispatch delivered 6d9bf0f3-77ee-4a1b-b1bd-795637677ac5 to worker[183] qsize 0
2021-05-03 22:47:24,769 DEBUG awx.main.dispatch task 6d9bf0f3-77ee-4a1b-b1bd-795637677ac5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:47:24,815 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:47:24,833 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:46:54.850572+00:00
2021-05-03 22:47:24,904 DEBUG awx.main.dispatch task 6d9bf0f3-77ee-4a1b-b1bd-795637677ac5 is finished
2021-05-03 22:47:34,777 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(dfad5508-1949-4193-9249-be7b5255207c, queue=awx_private_queue)
[2021-05-03 22:47:34,777: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(dfad5508-1949-4193-9249-be7b5255207c, queue=awx_private_queue)
2021-05-03 22:47:34,811 DEBUG awx.main.dispatch delivered dfad5508-1949-4193-9249-be7b5255207c to worker[184] qsize 0
2021-05-03 22:47:34,815 DEBUG awx.main.dispatch task dfad5508-1949-4193-9249-be7b5255207c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:47:34,818 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:47:34,872 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:47:34,959 DEBUG awx.main.dispatch task dfad5508-1949-4193-9249-be7b5255207c is finished
2021-05-03 22:47:54,479 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(cd6cf731-cce2-4a48-ad31-f0b0c73b396c, queue=dev-awx-0)
[2021-05-03 22:47:54,479: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(cd6cf731-cce2-4a48-ad31-f0b0c73b396c, queue=dev-awx-0)
2021-05-03 22:47:54,847 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(67555d5f-0b53-4822-a991-c0918839f52b, queue=dev-awx-0)
[2021-05-03 22:47:54,847: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(67555d5f-0b53-4822-a991-c0918839f52b, queue=dev-awx-0)
2021-05-03 22:47:54,864 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7e3c664f-6687-45c6-90c0-e9a64287c9bb, queue=awx_private_queue)
[2021-05-03 22:47:54,864: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7e3c664f-6687-45c6-90c0-e9a64287c9bb, queue=awx_private_queue)
2021-05-03 22:47:54,879 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a68fb77e-498c-4dab-b2c7-42c18518ecb2, queue=awx_private_queue)
[2021-05-03 22:47:54,879: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a68fb77e-498c-4dab-b2c7-42c18518ecb2, queue=awx_private_queue)
2021-05-03 22:47:54,882 DEBUG awx.main.dispatch delivered cd6cf731-cce2-4a48-ad31-f0b0c73b396c to worker[184] qsize 0
2021-05-03 22:47:54,886 DEBUG awx.main.dispatch delivered 67555d5f-0b53-4822-a991-c0918839f52b to worker[183] qsize 0
2021-05-03 22:47:54,888 DEBUG awx.main.dispatch delivered 7e3c664f-6687-45c6-90c0-e9a64287c9bb to worker[182] qsize 0
2021-05-03 22:47:54,890 DEBUG awx.main.dispatch task 67555d5f-0b53-4822-a991-c0918839f52b starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 22:47:54,887 DEBUG awx.main.dispatch task cd6cf731-cce2-4a48-ad31-f0b0c73b396c starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 22:47:54,895 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 22:47:54,893 DEBUG awx.main.dispatch task 7e3c664f-6687-45c6-90c0-e9a64287c9bb starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:47:54,910 DEBUG awx.main.dispatch delivered a68fb77e-498c-4dab-b2c7-42c18518ecb2 to worker[181] qsize 0
2021-05-03 22:47:54,917 DEBUG awx.main.dispatch task a68fb77e-498c-4dab-b2c7-42c18518ecb2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:47:54,921 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:47:54,927 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:47:54,930 DEBUG awx.main.dispatch task 67555d5f-0b53-4822-a991-c0918839f52b is finished
2021-05-03 22:47:54,945 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:47:24.815550+00:00
2021-05-03 22:47:55,002 DEBUG awx.main.dispatch task cd6cf731-cce2-4a48-ad31-f0b0c73b396c is finished
2021-05-03 22:47:55,034 DEBUG awx.main.dispatch task 7e3c664f-6687-45c6-90c0-e9a64287c9bb is finished
2021-05-03 22:47:55,086 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:47:55,304 DEBUG awx.main.dispatch task a68fb77e-498c-4dab-b2c7-42c18518ecb2 is finished
RESULT 2
OKREADY
2021-05-03 22:48:14,897 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(77e37a9c-6ecf-412e-bba4-2f575a06e92d, queue=awx_private_queue)
[2021-05-03 22:48:14,897: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(77e37a9c-6ecf-412e-bba4-2f575a06e92d, queue=awx_private_queue)
2021-05-03 22:48:14,964 DEBUG awx.main.dispatch delivered 77e37a9c-6ecf-412e-bba4-2f575a06e92d to worker[184] qsize 0
2021-05-03 22:48:14,973 DEBUG awx.main.dispatch task 77e37a9c-6ecf-412e-bba4-2f575a06e92d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:48:14,977 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:48:15,001 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:48:15,037 DEBUG awx.main.dispatch task 77e37a9c-6ecf-412e-bba4-2f575a06e92d is finished
2021-05-03 22:48:24,865 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(fc03cced-1664-47d4-8ae2-4edad4342126, queue=awx_private_queue)
[2021-05-03 22:48:24,865: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(fc03cced-1664-47d4-8ae2-4edad4342126, queue=awx_private_queue)
2021-05-03 22:48:24,885 DEBUG awx.main.dispatch delivered fc03cced-1664-47d4-8ae2-4edad4342126 to worker[182] qsize 0
2021-05-03 22:48:24,889 DEBUG awx.main.dispatch task fc03cced-1664-47d4-8ae2-4edad4342126 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:48:24,925 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:48:24,932 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:47:54.928235+00:00
2021-05-03 22:48:24,954 DEBUG awx.main.dispatch task fc03cced-1664-47d4-8ae2-4edad4342126 is finished
2021-05-03 22:48:34,892 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(606b8826-48b5-40c0-b40e-db4c29364314, queue=awx_private_queue)
[2021-05-03 22:48:34,892: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(606b8826-48b5-40c0-b40e-db4c29364314, queue=awx_private_queue)
2021-05-03 22:48:34,925 DEBUG awx.main.dispatch delivered 606b8826-48b5-40c0-b40e-db4c29364314 to worker[181] qsize 0
2021-05-03 22:48:34,934 DEBUG awx.main.dispatch task 606b8826-48b5-40c0-b40e-db4c29364314 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:48:34,938 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:48:34,977 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:48:35,025 DEBUG awx.main.dispatch task 606b8826-48b5-40c0-b40e-db4c29364314 is finished
2021-05-03 22:48:54,477 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(64b71770-c89d-4713-adec-c928edae9fb8, queue=dev-awx-0)
[2021-05-03 22:48:54,477: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(64b71770-c89d-4713-adec-c928edae9fb8, queue=dev-awx-0)
2021-05-03 22:48:54,542 DEBUG awx.main.dispatch delivered 64b71770-c89d-4713-adec-c928edae9fb8 to worker[181] qsize 0
2021-05-03 22:48:54,545 DEBUG awx.main.dispatch task 64b71770-c89d-4713-adec-c928edae9fb8 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 22:48:54,547 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 22:48:54,572 DEBUG awx.main.dispatch task 64b71770-c89d-4713-adec-c928edae9fb8 is finished
2021-05-03 22:48:54,847 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b1fd3ed2-e530-4aaa-8d7b-776bacb7985f, queue=dev-awx-0)
[2021-05-03 22:48:54,847: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b1fd3ed2-e530-4aaa-8d7b-776bacb7985f, queue=dev-awx-0)
2021-05-03 22:48:55,100 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(669c30c3-95be-49ec-bc59-7a56bd35be20, queue=awx_private_queue)
[2021-05-03 22:48:55,100: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(669c30c3-95be-49ec-bc59-7a56bd35be20, queue=awx_private_queue)
2021-05-03 22:48:55,099 DEBUG awx.main.dispatch delivered b1fd3ed2-e530-4aaa-8d7b-776bacb7985f to worker[182] qsize 0
2021-05-03 22:48:55,105 DEBUG awx.main.dispatch task b1fd3ed2-e530-4aaa-8d7b-776bacb7985f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 22:48:55,122 DEBUG awx.main.dispatch delivered 669c30c3-95be-49ec-bc59-7a56bd35be20 to worker[184] qsize 0
2021-05-03 22:48:55,123 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0ea65e16-fdc7-4f80-a9ec-224228f513ef, queue=awx_private_queue)
[2021-05-03 22:48:55,123: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0ea65e16-fdc7-4f80-a9ec-224228f513ef, queue=awx_private_queue)
2021-05-03 22:48:55,128 DEBUG awx.main.dispatch task b1fd3ed2-e530-4aaa-8d7b-776bacb7985f is finished
2021-05-03 22:48:55,126 DEBUG awx.main.dispatch task 669c30c3-95be-49ec-bc59-7a56bd35be20 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:48:55,141 DEBUG awx.main.dispatch delivered 0ea65e16-fdc7-4f80-a9ec-224228f513ef to worker[181] qsize 0
2021-05-03 22:48:55,143 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:48:55,144 DEBUG awx.main.dispatch task 0ea65e16-fdc7-4f80-a9ec-224228f513ef starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:48:55,146 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:48:55,148 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:48:24.927186+00:00
2021-05-03 22:48:55,160 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:48:55,173 DEBUG awx.main.dispatch task 669c30c3-95be-49ec-bc59-7a56bd35be20 is finished
2021-05-03 22:48:55,197 DEBUG awx.main.dispatch task 0ea65e16-fdc7-4f80-a9ec-224228f513ef is finished
RESULT 2
OKREADY
2021-05-03 22:49:15,134 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(89931401-5b6d-4336-b567-17d0ca475a22, queue=awx_private_queue)
[2021-05-03 22:49:15,134: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(89931401-5b6d-4336-b567-17d0ca475a22, queue=awx_private_queue)
2021-05-03 22:49:15,156 DEBUG awx.main.dispatch delivered 89931401-5b6d-4336-b567-17d0ca475a22 to worker[184] qsize 0
2021-05-03 22:49:15,157 DEBUG awx.main.dispatch task 89931401-5b6d-4336-b567-17d0ca475a22 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:49:15,159 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:49:15,179 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:49:15,214 DEBUG awx.main.dispatch task 89931401-5b6d-4336-b567-17d0ca475a22 is finished
2021-05-03 22:49:25,100 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d24e7fcb-a621-46a8-bbfd-85d7ed9c40bd, queue=awx_private_queue)
[2021-05-03 22:49:25,100: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d24e7fcb-a621-46a8-bbfd-85d7ed9c40bd, queue=awx_private_queue)
2021-05-03 22:49:25,115 DEBUG awx.main.dispatch delivered d24e7fcb-a621-46a8-bbfd-85d7ed9c40bd to worker[182] qsize 0
2021-05-03 22:49:25,118 DEBUG awx.main.dispatch task d24e7fcb-a621-46a8-bbfd-85d7ed9c40bd starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:49:25,128 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:49:25,131 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:48:55.143376+00:00
2021-05-03 22:49:25,147 DEBUG awx.main.dispatch task d24e7fcb-a621-46a8-bbfd-85d7ed9c40bd is finished
2021-05-03 22:49:35,142 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(128816c1-caef-4a92-95d3-cfd84f15f8e5, queue=awx_private_queue)
[2021-05-03 22:49:35,142: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(128816c1-caef-4a92-95d3-cfd84f15f8e5, queue=awx_private_queue)
2021-05-03 22:49:35,170 DEBUG awx.main.dispatch delivered 128816c1-caef-4a92-95d3-cfd84f15f8e5 to worker[183] qsize 0
2021-05-03 22:49:35,178 DEBUG awx.main.dispatch task 128816c1-caef-4a92-95d3-cfd84f15f8e5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:49:35,183 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:49:35,207 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:49:35,245 DEBUG awx.main.dispatch task 128816c1-caef-4a92-95d3-cfd84f15f8e5 is finished
2021-05-03 22:49:54,482 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(64d9fad6-c75e-4cb2-81c3-094aaa9e6f5d, queue=dev-awx-0)
[2021-05-03 22:49:54,482: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(64d9fad6-c75e-4cb2-81c3-094aaa9e6f5d, queue=dev-awx-0)
2021-05-03 22:49:54,546 DEBUG awx.main.dispatch delivered 64d9fad6-c75e-4cb2-81c3-094aaa9e6f5d to worker[184] qsize 0
2021-05-03 22:49:54,551 DEBUG awx.main.dispatch task 64d9fad6-c75e-4cb2-81c3-094aaa9e6f5d starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 22:49:54,554 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 22:49:54,589 DEBUG awx.main.dispatch task 64d9fad6-c75e-4cb2-81c3-094aaa9e6f5d is finished
2021-05-03 22:49:54,847 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(9f0d4e4f-7d8f-4b07-a03e-cbf626e0af91, queue=dev-awx-0)
[2021-05-03 22:49:54,847: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(9f0d4e4f-7d8f-4b07-a03e-cbf626e0af91, queue=dev-awx-0)
2021-05-03 22:49:55,056 DEBUG awx.main.dispatch delivered 9f0d4e4f-7d8f-4b07-a03e-cbf626e0af91 to worker[182] qsize 0
2021-05-03 22:49:55,059 DEBUG awx.main.dispatch task 9f0d4e4f-7d8f-4b07-a03e-cbf626e0af91 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 22:49:55,078 DEBUG awx.main.dispatch task 9f0d4e4f-7d8f-4b07-a03e-cbf626e0af91 is finished
2021-05-03 22:49:55,100 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(aab0b664-f8bf-497e-899e-dfefd14bdad5, queue=awx_private_queue)
[2021-05-03 22:49:55,100: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(aab0b664-f8bf-497e-899e-dfefd14bdad5, queue=awx_private_queue)
2021-05-03 22:49:55,122 DEBUG awx.main.dispatch delivered aab0b664-f8bf-497e-899e-dfefd14bdad5 to worker[181] qsize 0
2021-05-03 22:49:55,126 DEBUG awx.main.dispatch task aab0b664-f8bf-497e-899e-dfefd14bdad5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:49:55,136 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1e973e66-b5fa-475b-9044-5de3390f3b09, queue=awx_private_queue)
[2021-05-03 22:49:55,136: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1e973e66-b5fa-475b-9044-5de3390f3b09, queue=awx_private_queue)
2021-05-03 22:49:55,143 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:49:55,149 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:49:25.128269+00:00
2021-05-03 22:49:55,156 DEBUG awx.main.dispatch delivered 1e973e66-b5fa-475b-9044-5de3390f3b09 to worker[182] qsize 0
2021-05-03 22:49:55,158 DEBUG awx.main.dispatch task 1e973e66-b5fa-475b-9044-5de3390f3b09 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:49:55,160 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:49:55,166 DEBUG awx.main.dispatch task aab0b664-f8bf-497e-899e-dfefd14bdad5 is finished
2021-05-03 22:49:55,182 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:49:55,213 DEBUG awx.main.dispatch task 1e973e66-b5fa-475b-9044-5de3390f3b09 is finished
RESULT 2
OKREADY
2021-05-03 22:50:15,147 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(612a7906-8b3b-437c-b446-8a5104d3bbd3, queue=awx_private_queue)
[2021-05-03 22:50:15,147: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(612a7906-8b3b-437c-b446-8a5104d3bbd3, queue=awx_private_queue)
2021-05-03 22:50:15,162 DEBUG awx.main.dispatch delivered 612a7906-8b3b-437c-b446-8a5104d3bbd3 to worker[184] qsize 0
2021-05-03 22:50:15,165 DEBUG awx.main.dispatch task 612a7906-8b3b-437c-b446-8a5104d3bbd3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:50:15,168 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:50:15,194 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:50:15,228 DEBUG awx.main.dispatch task 612a7906-8b3b-437c-b446-8a5104d3bbd3 is finished
2021-05-03 22:50:25,100 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e4041352-ffe3-403b-bfd3-2f333411cbd5, queue=awx_private_queue)
[2021-05-03 22:50:25,100: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e4041352-ffe3-403b-bfd3-2f333411cbd5, queue=awx_private_queue)
2021-05-03 22:50:25,114 DEBUG awx.main.dispatch delivered e4041352-ffe3-403b-bfd3-2f333411cbd5 to worker[183] qsize 0
2021-05-03 22:50:25,116 DEBUG awx.main.dispatch task e4041352-ffe3-403b-bfd3-2f333411cbd5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:50:25,132 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:50:25,135 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:49:55.143576+00:00
2021-05-03 22:50:25,203 DEBUG awx.main.dispatch task e4041352-ffe3-403b-bfd3-2f333411cbd5 is finished
2021-05-03 22:50:35,147 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ee9621f7-5cac-4138-aeb9-b029c3412176, queue=awx_private_queue)
[2021-05-03 22:50:35,147: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ee9621f7-5cac-4138-aeb9-b029c3412176, queue=awx_private_queue)
2021-05-03 22:50:35,161 DEBUG awx.main.dispatch delivered ee9621f7-5cac-4138-aeb9-b029c3412176 to worker[182] qsize 0
2021-05-03 22:50:35,163 DEBUG awx.main.dispatch task ee9621f7-5cac-4138-aeb9-b029c3412176 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:50:35,164 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:50:35,180 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:50:35,206 DEBUG awx.main.dispatch task ee9621f7-5cac-4138-aeb9-b029c3412176 is finished
2021-05-03 22:50:54,497 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(086ec34d-3cdb-4448-8be1-b3f343837f20, queue=dev-awx-0)
[2021-05-03 22:50:54,497: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(086ec34d-3cdb-4448-8be1-b3f343837f20, queue=dev-awx-0)
2021-05-03 22:50:54,559 DEBUG awx.main.dispatch delivered 086ec34d-3cdb-4448-8be1-b3f343837f20 to worker[181] qsize 0
2021-05-03 22:50:54,565 DEBUG awx.main.dispatch task 086ec34d-3cdb-4448-8be1-b3f343837f20 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 22:50:54,570 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 22:50:54,600 DEBUG awx.main.dispatch task 086ec34d-3cdb-4448-8be1-b3f343837f20 is finished
2021-05-03 22:50:54,847 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2b704080-fbbb-4aee-928f-f82ab2c6de06, queue=dev-awx-0)
[2021-05-03 22:50:54,847: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2b704080-fbbb-4aee-928f-f82ab2c6de06, queue=dev-awx-0)
2021-05-03 22:50:54,862 DEBUG awx.main.dispatch delivered 2b704080-fbbb-4aee-928f-f82ab2c6de06 to worker[181] qsize 0
2021-05-03 22:50:54,864 DEBUG awx.main.dispatch task 2b704080-fbbb-4aee-928f-f82ab2c6de06 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 22:50:54,882 DEBUG awx.main.dispatch task 2b704080-fbbb-4aee-928f-f82ab2c6de06 is finished
2021-05-03 22:50:55,101 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(953c174c-b4e3-462f-af9d-79a6f1f85757, queue=awx_private_queue)
[2021-05-03 22:50:55,101: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(953c174c-b4e3-462f-af9d-79a6f1f85757, queue=awx_private_queue)
2021-05-03 22:50:55,305 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cef5a9a8-93f4-4828-b68f-d8273f297d84, queue=awx_private_queue)
2021-05-03 22:50:55,304 DEBUG awx.main.dispatch delivered 953c174c-b4e3-462f-af9d-79a6f1f85757 to worker[182] qsize 0
[2021-05-03 22:50:55,305: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cef5a9a8-93f4-4828-b68f-d8273f297d84, queue=awx_private_queue)
2021-05-03 22:50:55,308 DEBUG awx.main.dispatch task 953c174c-b4e3-462f-af9d-79a6f1f85757 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:50:55,319 DEBUG awx.main.dispatch delivered cef5a9a8-93f4-4828-b68f-d8273f297d84 to worker[184] qsize 0
2021-05-03 22:50:55,321 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:50:55,322 DEBUG awx.main.dispatch task cef5a9a8-93f4-4828-b68f-d8273f297d84 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:50:55,325 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:50:55,330 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:50:25.133079+00:00
2021-05-03 22:50:55,350 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:50:55,350 DEBUG awx.main.dispatch task 953c174c-b4e3-462f-af9d-79a6f1f85757 is finished
2021-05-03 22:50:55,405 DEBUG awx.main.dispatch task cef5a9a8-93f4-4828-b68f-d8273f297d84 is finished
RESULT 2
OKREADY
2021-05-03 22:51:15,315 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0aa08799-5224-401e-9cea-1deb11e88935, queue=awx_private_queue)
[2021-05-03 22:51:15,315: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0aa08799-5224-401e-9cea-1deb11e88935, queue=awx_private_queue)
2021-05-03 22:51:15,331 DEBUG awx.main.dispatch delivered 0aa08799-5224-401e-9cea-1deb11e88935 to worker[184] qsize 0
2021-05-03 22:51:15,334 DEBUG awx.main.dispatch task 0aa08799-5224-401e-9cea-1deb11e88935 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:51:15,336 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:51:15,354 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:51:15,387 DEBUG awx.main.dispatch task 0aa08799-5224-401e-9cea-1deb11e88935 is finished
2021-05-03 22:51:25,100 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(21a22166-a060-42a6-bcea-734b84734f4f, queue=awx_private_queue)
[2021-05-03 22:51:25,100: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(21a22166-a060-42a6-bcea-734b84734f4f, queue=awx_private_queue)
2021-05-03 22:51:25,116 DEBUG awx.main.dispatch delivered 21a22166-a060-42a6-bcea-734b84734f4f to worker[181] qsize 0
2021-05-03 22:51:25,118 DEBUG awx.main.dispatch task 21a22166-a060-42a6-bcea-734b84734f4f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:51:25,130 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:51:25,134 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:50:55.321966+00:00
2021-05-03 22:51:25,147 DEBUG awx.main.dispatch task 21a22166-a060-42a6-bcea-734b84734f4f is finished
2021-05-03 22:51:35,316 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(392a6a4b-1199-4a22-9b68-791c9d38a1b2, queue=awx_private_queue)
[2021-05-03 22:51:35,316: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(392a6a4b-1199-4a22-9b68-791c9d38a1b2, queue=awx_private_queue)
2021-05-03 22:51:35,337 DEBUG awx.main.dispatch delivered 392a6a4b-1199-4a22-9b68-791c9d38a1b2 to worker[182] qsize 0
2021-05-03 22:51:35,339 DEBUG awx.main.dispatch task 392a6a4b-1199-4a22-9b68-791c9d38a1b2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:51:35,340 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:51:35,361 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:51:35,393 DEBUG awx.main.dispatch task 392a6a4b-1199-4a22-9b68-791c9d38a1b2 is finished
2021-05-03 22:51:54,522 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e1c0104b-a635-40c1-a765-ed9f827bf2de, queue=dev-awx-0)
[2021-05-03 22:51:54,522: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e1c0104b-a635-40c1-a765-ed9f827bf2de, queue=dev-awx-0)
2021-05-03 22:51:54,596 DEBUG awx.main.dispatch delivered e1c0104b-a635-40c1-a765-ed9f827bf2de to worker[181] qsize 0
2021-05-03 22:51:54,610 DEBUG awx.main.dispatch task e1c0104b-a635-40c1-a765-ed9f827bf2de starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 22:51:54,614 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 22:51:54,639 DEBUG awx.main.dispatch task e1c0104b-a635-40c1-a765-ed9f827bf2de is finished
2021-05-03 22:51:54,848 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(53ccb1b5-e59d-4523-8f64-2cc64c4540c5, queue=dev-awx-0)
[2021-05-03 22:51:54,848: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(53ccb1b5-e59d-4523-8f64-2cc64c4540c5, queue=dev-awx-0)
2021-05-03 22:51:54,871 DEBUG awx.main.dispatch delivered 53ccb1b5-e59d-4523-8f64-2cc64c4540c5 to worker[181] qsize 0
2021-05-03 22:51:54,874 DEBUG awx.main.dispatch task 53ccb1b5-e59d-4523-8f64-2cc64c4540c5 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 22:51:54,886 DEBUG awx.main.dispatch task 53ccb1b5-e59d-4523-8f64-2cc64c4540c5 is finished
2021-05-03 22:51:55,100 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(99513ce5-263e-4b73-a810-f319f4e697b8, queue=awx_private_queue)
[2021-05-03 22:51:55,100: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(99513ce5-263e-4b73-a810-f319f4e697b8, queue=awx_private_queue)
2021-05-03 22:51:55,120 DEBUG awx.main.dispatch delivered 99513ce5-263e-4b73-a810-f319f4e697b8 to worker[182] qsize 0
2021-05-03 22:51:55,127 DEBUG awx.main.dispatch task 99513ce5-263e-4b73-a810-f319f4e697b8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:51:55,148 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:51:55,160 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:51:25.131093+00:00
2021-05-03 22:51:55,180 DEBUG awx.main.dispatch task 99513ce5-263e-4b73-a810-f319f4e697b8 is finished
2021-05-03 22:51:55,316 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2a9f5f6e-d5f3-4297-8441-45f416727dc8, queue=awx_private_queue)
[2021-05-03 22:51:55,316: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2a9f5f6e-d5f3-4297-8441-45f416727dc8, queue=awx_private_queue)
2021-05-03 22:51:55,519 DEBUG awx.main.dispatch delivered 2a9f5f6e-d5f3-4297-8441-45f416727dc8 to worker[184] qsize 0
2021-05-03 22:51:55,524 DEBUG awx.main.dispatch task 2a9f5f6e-d5f3-4297-8441-45f416727dc8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:51:55,527 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:51:55,549 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:51:55,587 DEBUG awx.main.dispatch task 2a9f5f6e-d5f3-4297-8441-45f416727dc8 is finished
RESULT 2
OKREADY
2021-05-03 22:52:15,329 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6af96cfc-7b80-4b9e-835b-fba78fb0157d, queue=awx_private_queue)
[2021-05-03 22:52:15,329: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6af96cfc-7b80-4b9e-835b-fba78fb0157d, queue=awx_private_queue)
2021-05-03 22:52:15,361 DEBUG awx.main.dispatch delivered 6af96cfc-7b80-4b9e-835b-fba78fb0157d to worker[181] qsize 0
2021-05-03 22:52:15,364 DEBUG awx.main.dispatch task 6af96cfc-7b80-4b9e-835b-fba78fb0157d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:52:15,365 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:52:15,385 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:52:15,419 DEBUG awx.main.dispatch task 6af96cfc-7b80-4b9e-835b-fba78fb0157d is finished
2021-05-03 22:52:25,101 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3e3a3f3b-3761-4e77-aa6a-4837cd12cd4a, queue=awx_private_queue)
[2021-05-03 22:52:25,101: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3e3a3f3b-3761-4e77-aa6a-4837cd12cd4a, queue=awx_private_queue)
2021-05-03 22:52:25,117 DEBUG awx.main.dispatch delivered 3e3a3f3b-3761-4e77-aa6a-4837cd12cd4a to worker[182] qsize 0
2021-05-03 22:52:25,119 DEBUG awx.main.dispatch task 3e3a3f3b-3761-4e77-aa6a-4837cd12cd4a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:52:25,128 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:52:25,130 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:51:55.148654+00:00
2021-05-03 22:52:25,143 DEBUG awx.main.dispatch task 3e3a3f3b-3761-4e77-aa6a-4837cd12cd4a is finished
2021-05-03 22:52:35,328 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(65b72ead-ec61-41a9-9563-865795fde89d, queue=awx_private_queue)
[2021-05-03 22:52:35,328: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(65b72ead-ec61-41a9-9563-865795fde89d, queue=awx_private_queue)
2021-05-03 22:52:35,343 DEBUG awx.main.dispatch delivered 65b72ead-ec61-41a9-9563-865795fde89d to worker[184] qsize 0
2021-05-03 22:52:35,345 DEBUG awx.main.dispatch task 65b72ead-ec61-41a9-9563-865795fde89d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:52:35,351 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:52:35,367 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:52:35,390 DEBUG awx.main.dispatch task 65b72ead-ec61-41a9-9563-865795fde89d is finished
2021-05-03 22:52:54,511 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(92f568ce-cb7f-4654-a2f5-fbc716b9b231, queue=dev-awx-0)
[2021-05-03 22:52:54,511: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(92f568ce-cb7f-4654-a2f5-fbc716b9b231, queue=dev-awx-0)
2021-05-03 22:52:54,543 DEBUG awx.main.dispatch delivered 92f568ce-cb7f-4654-a2f5-fbc716b9b231 to worker[182] qsize 0
2021-05-03 22:52:54,546 DEBUG awx.main.dispatch task 92f568ce-cb7f-4654-a2f5-fbc716b9b231 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 22:52:54,547 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 22:52:54,581 DEBUG awx.main.dispatch task 92f568ce-cb7f-4654-a2f5-fbc716b9b231 is finished
2021-05-03 22:52:54,847 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(bc698868-eae7-43e2-9818-f00f0721f5b4, queue=dev-awx-0)
[2021-05-03 22:52:54,847: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(bc698868-eae7-43e2-9818-f00f0721f5b4, queue=dev-awx-0)
2021-05-03 22:52:54,875 DEBUG awx.main.dispatch delivered bc698868-eae7-43e2-9818-f00f0721f5b4 to worker[181] qsize 0
2021-05-03 22:52:54,877 DEBUG awx.main.dispatch task bc698868-eae7-43e2-9818-f00f0721f5b4 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 22:52:55,095 DEBUG awx.main.dispatch task bc698868-eae7-43e2-9818-f00f0721f5b4 is finished
2021-05-03 22:52:55,101 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(639ade80-0b9f-4260-a7fe-90c4b11e6012, queue=awx_private_queue)
[2021-05-03 22:52:55,101: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(639ade80-0b9f-4260-a7fe-90c4b11e6012, queue=awx_private_queue)
2021-05-03 22:52:55,114 DEBUG awx.main.dispatch delivered 639ade80-0b9f-4260-a7fe-90c4b11e6012 to worker[182] qsize 0
2021-05-03 22:52:55,116 DEBUG awx.main.dispatch task 639ade80-0b9f-4260-a7fe-90c4b11e6012 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:52:55,124 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:52:55,127 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:52:25.128850+00:00
2021-05-03 22:52:55,139 DEBUG awx.main.dispatch task 639ade80-0b9f-4260-a7fe-90c4b11e6012 is finished
2021-05-03 22:52:55,329 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a2ce5838-7da0-4c86-86ff-1dc73a59cfb9, queue=awx_private_queue)
[2021-05-03 22:52:55,329: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a2ce5838-7da0-4c86-86ff-1dc73a59cfb9, queue=awx_private_queue)
2021-05-03 22:52:55,350 DEBUG awx.main.dispatch delivered a2ce5838-7da0-4c86-86ff-1dc73a59cfb9 to worker[183] qsize 0
2021-05-03 22:52:55,354 DEBUG awx.main.dispatch task a2ce5838-7da0-4c86-86ff-1dc73a59cfb9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:52:55,359 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:52:55,380 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:52:55,406 DEBUG awx.main.dispatch task a2ce5838-7da0-4c86-86ff-1dc73a59cfb9 is finished
RESULT 2
OKREADY
2021-05-03 22:53:15,342 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(66e51148-7ac6-47ee-98d0-9be143ccf461, queue=awx_private_queue)
[2021-05-03 22:53:15,342: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(66e51148-7ac6-47ee-98d0-9be143ccf461, queue=awx_private_queue)
2021-05-03 22:53:15,378 DEBUG awx.main.dispatch delivered 66e51148-7ac6-47ee-98d0-9be143ccf461 to worker[182] qsize 0
2021-05-03 22:53:15,387 DEBUG awx.main.dispatch task 66e51148-7ac6-47ee-98d0-9be143ccf461 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:53:15,391 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:53:15,420 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:53:15,449 DEBUG awx.main.dispatch task 66e51148-7ac6-47ee-98d0-9be143ccf461 is finished
2021-05-03 22:53:25,102 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4bc5b8f2-6fad-485f-b4a5-7f4a6630a934, queue=awx_private_queue)
[2021-05-03 22:53:25,102: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4bc5b8f2-6fad-485f-b4a5-7f4a6630a934, queue=awx_private_queue)
2021-05-03 22:53:25,121 DEBUG awx.main.dispatch delivered 4bc5b8f2-6fad-485f-b4a5-7f4a6630a934 to worker[182] qsize 0
2021-05-03 22:53:25,126 DEBUG awx.main.dispatch task 4bc5b8f2-6fad-485f-b4a5-7f4a6630a934 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:53:25,139 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:53:25,143 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:52:55.124289+00:00
2021-05-03 22:53:25,158 DEBUG awx.main.dispatch task 4bc5b8f2-6fad-485f-b4a5-7f4a6630a934 is finished
2021-05-03 22:53:35,338 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(08293419-955f-4f3f-a348-4df8cfe9f0ff, queue=awx_private_queue)
[2021-05-03 22:53:35,338: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(08293419-955f-4f3f-a348-4df8cfe9f0ff, queue=awx_private_queue)
2021-05-03 22:53:35,356 DEBUG awx.main.dispatch delivered 08293419-955f-4f3f-a348-4df8cfe9f0ff to worker[182] qsize 0
2021-05-03 22:53:35,359 DEBUG awx.main.dispatch task 08293419-955f-4f3f-a348-4df8cfe9f0ff starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:53:35,360 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:53:35,379 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:53:35,404 DEBUG awx.main.dispatch task 08293419-955f-4f3f-a348-4df8cfe9f0ff is finished
2021-05-03 22:53:54,522 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(194dad22-fed1-4650-adc3-41b6637191b3, queue=dev-awx-0)
[2021-05-03 22:53:54,522: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(194dad22-fed1-4650-adc3-41b6637191b3, queue=dev-awx-0)
2021-05-03 22:53:54,583 DEBUG awx.main.dispatch delivered 194dad22-fed1-4650-adc3-41b6637191b3 to worker[183] qsize 0
2021-05-03 22:53:54,588 DEBUG awx.main.dispatch task 194dad22-fed1-4650-adc3-41b6637191b3 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 22:53:54,592 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 22:53:54,632 DEBUG awx.main.dispatch task 194dad22-fed1-4650-adc3-41b6637191b3 is finished
2021-05-03 22:53:54,847 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(33fde6cd-d074-4658-aaf6-d27525ee519a, queue=dev-awx-0)
[2021-05-03 22:53:54,847: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(33fde6cd-d074-4658-aaf6-d27525ee519a, queue=dev-awx-0)
2021-05-03 22:53:54,877 DEBUG awx.main.dispatch delivered 33fde6cd-d074-4658-aaf6-d27525ee519a to worker[181] qsize 0
2021-05-03 22:53:54,881 DEBUG awx.main.dispatch task 33fde6cd-d074-4658-aaf6-d27525ee519a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 22:53:55,093 DEBUG awx.main.dispatch task 33fde6cd-d074-4658-aaf6-d27525ee519a is finished
2021-05-03 22:53:55,101 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4f7b8e2e-d8f1-4ea7-82de-aa70ca61cf88, queue=awx_private_queue)
[2021-05-03 22:53:55,101: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4f7b8e2e-d8f1-4ea7-82de-aa70ca61cf88, queue=awx_private_queue)
2021-05-03 22:53:55,117 DEBUG awx.main.dispatch delivered 4f7b8e2e-d8f1-4ea7-82de-aa70ca61cf88 to worker[181] qsize 0
2021-05-03 22:53:55,120 DEBUG awx.main.dispatch task 4f7b8e2e-d8f1-4ea7-82de-aa70ca61cf88 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:53:55,130 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:53:55,132 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:53:25.139759+00:00
2021-05-03 22:53:55,144 DEBUG awx.main.dispatch task 4f7b8e2e-d8f1-4ea7-82de-aa70ca61cf88 is finished
2021-05-03 22:53:55,337 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8ec881a6-030a-4af9-b8f4-72a9f137216b, queue=awx_private_queue)
[2021-05-03 22:53:55,337: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8ec881a6-030a-4af9-b8f4-72a9f137216b, queue=awx_private_queue)
2021-05-03 22:53:55,358 DEBUG awx.main.dispatch delivered 8ec881a6-030a-4af9-b8f4-72a9f137216b to worker[181] qsize 0
2021-05-03 22:53:55,361 DEBUG awx.main.dispatch task 8ec881a6-030a-4af9-b8f4-72a9f137216b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:53:55,362 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:53:55,380 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:53:55,407 DEBUG awx.main.dispatch task 8ec881a6-030a-4af9-b8f4-72a9f137216b is finished
RESULT 2
OKREADY
2021-05-03 22:54:15,348 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(aca3cb4b-1cb5-4f02-8d67-d0faba8f5cc8, queue=awx_private_queue)
[2021-05-03 22:54:15,348: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(aca3cb4b-1cb5-4f02-8d67-d0faba8f5cc8, queue=awx_private_queue)
2021-05-03 22:54:15,363 DEBUG awx.main.dispatch delivered aca3cb4b-1cb5-4f02-8d67-d0faba8f5cc8 to worker[182] qsize 0
2021-05-03 22:54:15,365 DEBUG awx.main.dispatch task aca3cb4b-1cb5-4f02-8d67-d0faba8f5cc8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:54:15,367 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:54:15,381 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:54:15,405 DEBUG awx.main.dispatch task aca3cb4b-1cb5-4f02-8d67-d0faba8f5cc8 is finished
2021-05-03 22:54:25,101 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ed6a2231-19e0-44dc-a709-7fdc67f784e6, queue=awx_private_queue)
[2021-05-03 22:54:25,101: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ed6a2231-19e0-44dc-a709-7fdc67f784e6, queue=awx_private_queue)
2021-05-03 22:54:25,118 DEBUG awx.main.dispatch delivered ed6a2231-19e0-44dc-a709-7fdc67f784e6 to worker[182] qsize 0
2021-05-03 22:54:25,121 DEBUG awx.main.dispatch task ed6a2231-19e0-44dc-a709-7fdc67f784e6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:54:25,130 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:54:25,132 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:53:55.130377+00:00
2021-05-03 22:54:25,143 DEBUG awx.main.dispatch task ed6a2231-19e0-44dc-a709-7fdc67f784e6 is finished
2021-05-03 22:54:35,348 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9e0ae78e-3b6d-4aa6-9b41-75a66937484e, queue=awx_private_queue)
[2021-05-03 22:54:35,348: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9e0ae78e-3b6d-4aa6-9b41-75a66937484e, queue=awx_private_queue)
2021-05-03 22:54:35,366 DEBUG awx.main.dispatch delivered 9e0ae78e-3b6d-4aa6-9b41-75a66937484e to worker[181] qsize 0
2021-05-03 22:54:35,368 DEBUG awx.main.dispatch task 9e0ae78e-3b6d-4aa6-9b41-75a66937484e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:54:35,371 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:54:35,389 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:54:35,419 DEBUG awx.main.dispatch task 9e0ae78e-3b6d-4aa6-9b41-75a66937484e is finished
2021-05-03 22:54:54,534 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(aed37181-08c1-4206-82cd-73a5e363b6dc, queue=dev-awx-0)
[2021-05-03 22:54:54,534: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(aed37181-08c1-4206-82cd-73a5e363b6dc, queue=dev-awx-0)
2021-05-03 22:54:54,580 DEBUG awx.main.dispatch delivered aed37181-08c1-4206-82cd-73a5e363b6dc to worker[182] qsize 0
2021-05-03 22:54:54,585 DEBUG awx.main.dispatch task aed37181-08c1-4206-82cd-73a5e363b6dc starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 22:54:54,590 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 22:54:54,622 DEBUG awx.main.dispatch task aed37181-08c1-4206-82cd-73a5e363b6dc is finished
2021-05-03 22:54:54,848 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8ad65290-102d-4fae-abd8-9101e182eabf, queue=dev-awx-0)
[2021-05-03 22:54:54,848: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8ad65290-102d-4fae-abd8-9101e182eabf, queue=dev-awx-0)
2021-05-03 22:54:54,879 DEBUG awx.main.dispatch delivered 8ad65290-102d-4fae-abd8-9101e182eabf to worker[183] qsize 0
2021-05-03 22:54:55,101 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d6ea1862-233e-4e58-b98d-11bf6d52d6d8, queue=awx_private_queue)
[2021-05-03 22:54:55,101: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d6ea1862-233e-4e58-b98d-11bf6d52d6d8, queue=awx_private_queue)
2021-05-03 22:54:55,114 DEBUG awx.main.dispatch delivered d6ea1862-233e-4e58-b98d-11bf6d52d6d8 to worker[184] qsize 0
2021-05-03 22:54:54,883 DEBUG awx.main.dispatch task 8ad65290-102d-4fae-abd8-9101e182eabf starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 22:54:55,152 DEBUG awx.main.dispatch task 8ad65290-102d-4fae-abd8-9101e182eabf is finished
2021-05-03 22:54:55,118 DEBUG awx.main.dispatch task d6ea1862-233e-4e58-b98d-11bf6d52d6d8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:54:55,325 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:54:55,328 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:54:25.130426+00:00
2021-05-03 22:54:55,340 DEBUG awx.main.dispatch task d6ea1862-233e-4e58-b98d-11bf6d52d6d8 is finished
2021-05-03 22:54:55,349 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a5392339-1d52-4a13-b5e8-731932b9fbb1, queue=awx_private_queue)
[2021-05-03 22:54:55,349: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a5392339-1d52-4a13-b5e8-731932b9fbb1, queue=awx_private_queue)
2021-05-03 22:54:55,372 DEBUG awx.main.dispatch delivered a5392339-1d52-4a13-b5e8-731932b9fbb1 to worker[183] qsize 0
2021-05-03 22:54:55,375 DEBUG awx.main.dispatch task a5392339-1d52-4a13-b5e8-731932b9fbb1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:54:55,377 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:54:55,397 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:54:55,424 DEBUG awx.main.dispatch task a5392339-1d52-4a13-b5e8-731932b9fbb1 is finished
RESULT 2
OKREADY
2021-05-03 22:55:15,361 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(05056edd-2eea-4b37-bf00-775c92cab601, queue=awx_private_queue)
[2021-05-03 22:55:15,361: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(05056edd-2eea-4b37-bf00-775c92cab601, queue=awx_private_queue)
2021-05-03 22:55:15,377 DEBUG awx.main.dispatch delivered 05056edd-2eea-4b37-bf00-775c92cab601 to worker[181] qsize 0
2021-05-03 22:55:15,381 DEBUG awx.main.dispatch task 05056edd-2eea-4b37-bf00-775c92cab601 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:55:15,387 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:55:15,423 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:55:15,466 DEBUG awx.main.dispatch task 05056edd-2eea-4b37-bf00-775c92cab601 is finished
2021-05-03 22:55:25,102 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3f1e80df-360a-4a09-a8a3-9317aa5c974e, queue=awx_private_queue)
[2021-05-03 22:55:25,102: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3f1e80df-360a-4a09-a8a3-9317aa5c974e, queue=awx_private_queue)
2021-05-03 22:55:25,116 DEBUG awx.main.dispatch delivered 3f1e80df-360a-4a09-a8a3-9317aa5c974e to worker[182] qsize 0
2021-05-03 22:55:25,119 DEBUG awx.main.dispatch task 3f1e80df-360a-4a09-a8a3-9317aa5c974e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:55:25,130 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:55:25,133 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:54:55.325791+00:00
2021-05-03 22:55:25,156 DEBUG awx.main.dispatch task 3f1e80df-360a-4a09-a8a3-9317aa5c974e is finished
2021-05-03 22:55:35,360 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(117b0aca-1397-452e-866f-aabe44f1e864, queue=awx_private_queue)
[2021-05-03 22:55:35,360: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(117b0aca-1397-452e-866f-aabe44f1e864, queue=awx_private_queue)
2021-05-03 22:55:35,375 DEBUG awx.main.dispatch delivered 117b0aca-1397-452e-866f-aabe44f1e864 to worker[184] qsize 0
2021-05-03 22:55:35,378 DEBUG awx.main.dispatch task 117b0aca-1397-452e-866f-aabe44f1e864 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:55:35,380 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:55:35,396 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:55:35,422 DEBUG awx.main.dispatch task 117b0aca-1397-452e-866f-aabe44f1e864 is finished
2021-05-03 22:55:54,480 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(4b23b62b-c263-4e7d-b3b2-446e7dd537b5, queue=dev-awx-0)
[2021-05-03 22:55:54,480: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(4b23b62b-c263-4e7d-b3b2-446e7dd537b5, queue=dev-awx-0)
2021-05-03 22:55:54,497 DEBUG awx.main.dispatch delivered 4b23b62b-c263-4e7d-b3b2-446e7dd537b5 to worker[182] qsize 0
2021-05-03 22:55:54,499 DEBUG awx.main.dispatch task 4b23b62b-c263-4e7d-b3b2-446e7dd537b5 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-05-03 22:55:54,500 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-05-03 22:55:54,521 DEBUG awx.main.dispatch task 4b23b62b-c263-4e7d-b3b2-446e7dd537b5 is finished
2021-05-03 22:55:54,529 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(55e5c5f7-ec22-4a1c-9cee-0eeec424a2ad, queue=dev-awx-0)
[2021-05-03 22:55:54,529: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(55e5c5f7-ec22-4a1c-9cee-0eeec424a2ad, queue=dev-awx-0)
2021-05-03 22:55:54,557 DEBUG awx.main.dispatch delivered 55e5c5f7-ec22-4a1c-9cee-0eeec424a2ad to worker[184] qsize 0
2021-05-03 22:55:54,559 DEBUG awx.main.dispatch task 55e5c5f7-ec22-4a1c-9cee-0eeec424a2ad starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 22:55:54,560 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 22:55:54,580 DEBUG awx.main.dispatch task 55e5c5f7-ec22-4a1c-9cee-0eeec424a2ad is finished
2021-05-03 22:55:54,847 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(0efa28a3-d457-473c-a34d-dff3c9a3ac11, queue=dev-awx-0)
[2021-05-03 22:55:54,847: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(0efa28a3-d457-473c-a34d-dff3c9a3ac11, queue=dev-awx-0)
2021-05-03 22:55:54,886 DEBUG awx.main.dispatch delivered 0efa28a3-d457-473c-a34d-dff3c9a3ac11 to worker[181] qsize 0
2021-05-03 22:55:54,888 DEBUG awx.main.dispatch task 0efa28a3-d457-473c-a34d-dff3c9a3ac11 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 22:55:54,905 DEBUG awx.main.dispatch task 0efa28a3-d457-473c-a34d-dff3c9a3ac11 is finished
2021-05-03 22:55:55,101 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8f851692-c2e3-4da4-88e7-947c2bca8a69, queue=awx_private_queue)
[2021-05-03 22:55:55,101: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8f851692-c2e3-4da4-88e7-947c2bca8a69, queue=awx_private_queue)
2021-05-03 22:55:55,115 DEBUG awx.main.dispatch delivered 8f851692-c2e3-4da4-88e7-947c2bca8a69 to worker[183] qsize 0
2021-05-03 22:55:55,117 DEBUG awx.main.dispatch task 8f851692-c2e3-4da4-88e7-947c2bca8a69 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:55:55,286 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:55:55,288 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:55:25.130580+00:00
2021-05-03 22:55:55,302 DEBUG awx.main.dispatch task 8f851692-c2e3-4da4-88e7-947c2bca8a69 is finished
2021-05-03 22:55:55,361 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(87fdb370-9dc4-434e-bc31-5c9438b3c625, queue=awx_private_queue)
[2021-05-03 22:55:55,361: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(87fdb370-9dc4-434e-bc31-5c9438b3c625, queue=awx_private_queue)
2021-05-03 22:55:55,387 DEBUG awx.main.dispatch delivered 87fdb370-9dc4-434e-bc31-5c9438b3c625 to worker[181] qsize 0
2021-05-03 22:55:55,389 DEBUG awx.main.dispatch task 87fdb370-9dc4-434e-bc31-5c9438b3c625 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:55:55,390 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:55:55,403 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:55:55,429 DEBUG awx.main.dispatch task 87fdb370-9dc4-434e-bc31-5c9438b3c625 is finished
RESULT 2
OKREADY
2021-05-03 22:56:15,376 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(838634db-332a-4c6a-9664-bf2aff868b18, queue=awx_private_queue)
[2021-05-03 22:56:15,376: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(838634db-332a-4c6a-9664-bf2aff868b18, queue=awx_private_queue)
2021-05-03 22:56:15,412 DEBUG awx.main.dispatch delivered 838634db-332a-4c6a-9664-bf2aff868b18 to worker[181] qsize 0
2021-05-03 22:56:15,420 DEBUG awx.main.dispatch task 838634db-332a-4c6a-9664-bf2aff868b18 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:56:15,424 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:56:15,459 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:56:15,496 DEBUG awx.main.dispatch task 838634db-332a-4c6a-9664-bf2aff868b18 is finished
2021-05-03 22:56:25,103 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(274f1fdd-a18d-4b11-9771-6b3a8b2d73db, queue=awx_private_queue)
[2021-05-03 22:56:25,103: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(274f1fdd-a18d-4b11-9771-6b3a8b2d73db, queue=awx_private_queue)
2021-05-03 22:56:25,123 DEBUG awx.main.dispatch delivered 274f1fdd-a18d-4b11-9771-6b3a8b2d73db to worker[181] qsize 0
2021-05-03 22:56:25,127 DEBUG awx.main.dispatch task 274f1fdd-a18d-4b11-9771-6b3a8b2d73db starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:56:25,144 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:56:25,150 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:55:55.286204+00:00
2021-05-03 22:56:25,169 DEBUG awx.main.dispatch task 274f1fdd-a18d-4b11-9771-6b3a8b2d73db is finished
2021-05-03 22:56:35,370 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e30e6c23-6a15-4a19-b20c-875e4908896b, queue=awx_private_queue)
[2021-05-03 22:56:35,370: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e30e6c23-6a15-4a19-b20c-875e4908896b, queue=awx_private_queue)
2021-05-03 22:56:35,385 DEBUG awx.main.dispatch delivered e30e6c23-6a15-4a19-b20c-875e4908896b to worker[181] qsize 0
2021-05-03 22:56:35,388 DEBUG awx.main.dispatch task e30e6c23-6a15-4a19-b20c-875e4908896b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:56:35,389 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:56:35,403 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:56:35,426 DEBUG awx.main.dispatch task e30e6c23-6a15-4a19-b20c-875e4908896b is finished
2021-05-03 22:56:54,530 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(0fae1051-2f4d-4c29-82d0-47e1cdb0d815, queue=dev-awx-0)
[2021-05-03 22:56:54,530: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(0fae1051-2f4d-4c29-82d0-47e1cdb0d815, queue=dev-awx-0)
2021-05-03 22:56:54,565 DEBUG awx.main.dispatch delivered 0fae1051-2f4d-4c29-82d0-47e1cdb0d815 to worker[182] qsize 0
2021-05-03 22:56:54,570 DEBUG awx.main.dispatch task 0fae1051-2f4d-4c29-82d0-47e1cdb0d815 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 22:56:54,574 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 22:56:54,600 DEBUG awx.main.dispatch task 0fae1051-2f4d-4c29-82d0-47e1cdb0d815 is finished
2021-05-03 22:56:54,848 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(591148f9-630d-402a-9a7d-afb4e944e56e, queue=dev-awx-0)
[2021-05-03 22:56:54,848: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(591148f9-630d-402a-9a7d-afb4e944e56e, queue=dev-awx-0)
2021-05-03 22:56:54,865 DEBUG awx.main.dispatch delivered 591148f9-630d-402a-9a7d-afb4e944e56e to worker[183] qsize 0
2021-05-03 22:56:54,870 DEBUG awx.main.dispatch task 591148f9-630d-402a-9a7d-afb4e944e56e starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 22:56:54,886 DEBUG awx.main.dispatch task 591148f9-630d-402a-9a7d-afb4e944e56e is finished
2021-05-03 22:56:55,101 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(05b6cb55-c37d-4fec-b9cf-06d21ac8f834, queue=awx_private_queue)
[2021-05-03 22:56:55,101: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(05b6cb55-c37d-4fec-b9cf-06d21ac8f834, queue=awx_private_queue)
2021-05-03 22:56:55,116 DEBUG awx.main.dispatch delivered 05b6cb55-c37d-4fec-b9cf-06d21ac8f834 to worker[183] qsize 0
2021-05-03 22:56:55,119 DEBUG awx.main.dispatch task 05b6cb55-c37d-4fec-b9cf-06d21ac8f834 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:56:55,357 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:56:55,360 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:56:25.144683+00:00
2021-05-03 22:56:55,370 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(125000cf-b424-463d-bd5b-8d270e113a96, queue=awx_private_queue)
[2021-05-03 22:56:55,370: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(125000cf-b424-463d-bd5b-8d270e113a96, queue=awx_private_queue)
2021-05-03 22:56:55,375 DEBUG awx.main.dispatch task 05b6cb55-c37d-4fec-b9cf-06d21ac8f834 is finished
2021-05-03 22:56:55,385 DEBUG awx.main.dispatch delivered 125000cf-b424-463d-bd5b-8d270e113a96 to worker[182] qsize 0
2021-05-03 22:56:55,388 DEBUG awx.main.dispatch task 125000cf-b424-463d-bd5b-8d270e113a96 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:56:55,389 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:56:55,409 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:56:55,442 DEBUG awx.main.dispatch task 125000cf-b424-463d-bd5b-8d270e113a96 is finished
RESULT 2
OKREADY
2021-05-03 22:57:15,387 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ec0189cb-76cd-437d-aa0f-68092cc761bc, queue=awx_private_queue)
[2021-05-03 22:57:15,387: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ec0189cb-76cd-437d-aa0f-68092cc761bc, queue=awx_private_queue)
2021-05-03 22:57:15,427 DEBUG awx.main.dispatch delivered ec0189cb-76cd-437d-aa0f-68092cc761bc to worker[181] qsize 0
2021-05-03 22:57:15,434 DEBUG awx.main.dispatch task ec0189cb-76cd-437d-aa0f-68092cc761bc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:57:15,440 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:57:15,474 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:57:15,503 DEBUG awx.main.dispatch task ec0189cb-76cd-437d-aa0f-68092cc761bc is finished
2021-05-03 22:57:25,103 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(278ac539-2304-4cc0-b0bf-70e3823e3e98, queue=awx_private_queue)
[2021-05-03 22:57:25,103: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(278ac539-2304-4cc0-b0bf-70e3823e3e98, queue=awx_private_queue)
2021-05-03 22:57:25,123 DEBUG awx.main.dispatch delivered 278ac539-2304-4cc0-b0bf-70e3823e3e98 to worker[182] qsize 0
2021-05-03 22:57:25,127 DEBUG awx.main.dispatch task 278ac539-2304-4cc0-b0bf-70e3823e3e98 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:57:25,144 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:57:25,151 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:56:55.357406+00:00
2021-05-03 22:57:25,172 DEBUG awx.main.dispatch task 278ac539-2304-4cc0-b0bf-70e3823e3e98 is finished
2021-05-03 22:57:35,382 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(248f0b70-c135-4ded-a92f-f546a04b0c2b, queue=awx_private_queue)
[2021-05-03 22:57:35,382: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(248f0b70-c135-4ded-a92f-f546a04b0c2b, queue=awx_private_queue)
2021-05-03 22:57:35,399 DEBUG awx.main.dispatch delivered 248f0b70-c135-4ded-a92f-f546a04b0c2b to worker[182] qsize 0
2021-05-03 22:57:35,401 DEBUG awx.main.dispatch task 248f0b70-c135-4ded-a92f-f546a04b0c2b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:57:35,403 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:57:35,419 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:57:35,458 DEBUG awx.main.dispatch task 248f0b70-c135-4ded-a92f-f546a04b0c2b is finished
2021-05-03 22:57:54,539 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(97efb2c6-2b30-4b1d-af0b-d6613632e1e8, queue=dev-awx-0)
[2021-05-03 22:57:54,539: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(97efb2c6-2b30-4b1d-af0b-d6613632e1e8, queue=dev-awx-0)
2021-05-03 22:57:54,593 DEBUG awx.main.dispatch delivered 97efb2c6-2b30-4b1d-af0b-d6613632e1e8 to worker[181] qsize 0
2021-05-03 22:57:54,595 DEBUG awx.main.dispatch task 97efb2c6-2b30-4b1d-af0b-d6613632e1e8 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 22:57:54,597 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 22:57:54,620 DEBUG awx.main.dispatch task 97efb2c6-2b30-4b1d-af0b-d6613632e1e8 is finished
2021-05-03 22:57:54,848 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(01bb1399-44c4-4126-822e-0de834463d9e, queue=dev-awx-0)
[2021-05-03 22:57:54,848: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(01bb1399-44c4-4126-822e-0de834463d9e, queue=dev-awx-0)
2021-05-03 22:57:54,872 DEBUG awx.main.dispatch delivered 01bb1399-44c4-4126-822e-0de834463d9e to worker[184] qsize 0
2021-05-03 22:57:54,877 DEBUG awx.main.dispatch task 01bb1399-44c4-4126-822e-0de834463d9e starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 22:57:54,907 DEBUG awx.main.dispatch task 01bb1399-44c4-4126-822e-0de834463d9e is finished
2021-05-03 22:57:55,102 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b50c1209-1fca-4646-9833-be12434cc5fb, queue=awx_private_queue)
[2021-05-03 22:57:55,102: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b50c1209-1fca-4646-9833-be12434cc5fb, queue=awx_private_queue)
2021-05-03 22:57:55,143 DEBUG awx.main.dispatch delivered b50c1209-1fca-4646-9833-be12434cc5fb to worker[181] qsize 0
2021-05-03 22:57:55,145 DEBUG awx.main.dispatch task b50c1209-1fca-4646-9833-be12434cc5fb starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:57:55,152 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:57:55,154 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:57:25.145042+00:00
2021-05-03 22:57:55,166 DEBUG awx.main.dispatch task b50c1209-1fca-4646-9833-be12434cc5fb is finished
2021-05-03 22:57:55,382 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cb409f9a-ff73-48b8-847f-3c2a7581022b, queue=awx_private_queue)
[2021-05-03 22:57:55,382: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cb409f9a-ff73-48b8-847f-3c2a7581022b, queue=awx_private_queue)
2021-05-03 22:57:55,624 DEBUG awx.main.dispatch delivered cb409f9a-ff73-48b8-847f-3c2a7581022b to worker[182] qsize 0
2021-05-03 22:57:55,626 DEBUG awx.main.dispatch task cb409f9a-ff73-48b8-847f-3c2a7581022b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:57:55,628 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:57:55,650 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:57:55,696 DEBUG awx.main.dispatch task cb409f9a-ff73-48b8-847f-3c2a7581022b is finished
RESULT 2
OKREADY
2021-05-03 22:58:15,401 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(81705486-19b6-460c-8392-59a97f06c2d8, queue=awx_private_queue)
[2021-05-03 22:58:15,401: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(81705486-19b6-460c-8392-59a97f06c2d8, queue=awx_private_queue)
2021-05-03 22:58:15,431 DEBUG awx.main.dispatch delivered 81705486-19b6-460c-8392-59a97f06c2d8 to worker[181] qsize 0
2021-05-03 22:58:15,438 DEBUG awx.main.dispatch task 81705486-19b6-460c-8392-59a97f06c2d8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:58:15,442 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:58:15,469 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:58:15,496 DEBUG awx.main.dispatch task 81705486-19b6-460c-8392-59a97f06c2d8 is finished
2021-05-03 22:58:25,102 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(111e75a5-ce8f-44da-9293-ad74245d6b85, queue=awx_private_queue)
[2021-05-03 22:58:25,102: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(111e75a5-ce8f-44da-9293-ad74245d6b85, queue=awx_private_queue)
2021-05-03 22:58:25,116 DEBUG awx.main.dispatch delivered 111e75a5-ce8f-44da-9293-ad74245d6b85 to worker[181] qsize 0
2021-05-03 22:58:25,117 DEBUG awx.main.dispatch task 111e75a5-ce8f-44da-9293-ad74245d6b85 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:58:25,128 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:58:25,131 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:57:55.152710+00:00
2021-05-03 22:58:25,145 DEBUG awx.main.dispatch task 111e75a5-ce8f-44da-9293-ad74245d6b85 is finished
2021-05-03 22:58:35,397 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(988a52bb-54dc-4558-bea1-918c250d3956, queue=awx_private_queue)
[2021-05-03 22:58:35,397: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(988a52bb-54dc-4558-bea1-918c250d3956, queue=awx_private_queue)
2021-05-03 22:58:35,410 DEBUG awx.main.dispatch delivered 988a52bb-54dc-4558-bea1-918c250d3956 to worker[183] qsize 0
2021-05-03 22:58:35,418 DEBUG awx.main.dispatch task 988a52bb-54dc-4558-bea1-918c250d3956 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:58:35,422 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:58:35,445 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:58:35,473 DEBUG awx.main.dispatch task 988a52bb-54dc-4558-bea1-918c250d3956 is finished
2021-05-03 22:58:54,556 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(4c80fddf-91db-492d-a1dd-2e017940f29f, queue=dev-awx-0)
[2021-05-03 22:58:54,556: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(4c80fddf-91db-492d-a1dd-2e017940f29f, queue=dev-awx-0)
2021-05-03 22:58:54,601 DEBUG awx.main.dispatch delivered 4c80fddf-91db-492d-a1dd-2e017940f29f to worker[184] qsize 0
2021-05-03 22:58:54,607 DEBUG awx.main.dispatch task 4c80fddf-91db-492d-a1dd-2e017940f29f starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 22:58:54,613 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 22:58:54,648 DEBUG awx.main.dispatch task 4c80fddf-91db-492d-a1dd-2e017940f29f is finished
2021-05-03 22:58:54,848 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5f59eb71-6b76-462c-a639-3013662f2c6e, queue=dev-awx-0)
[2021-05-03 22:58:54,848: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5f59eb71-6b76-462c-a639-3013662f2c6e, queue=dev-awx-0)
2021-05-03 22:58:54,864 DEBUG awx.main.dispatch delivered 5f59eb71-6b76-462c-a639-3013662f2c6e to worker[184] qsize 0
2021-05-03 22:58:54,868 DEBUG awx.main.dispatch task 5f59eb71-6b76-462c-a639-3013662f2c6e starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 22:58:54,891 DEBUG awx.main.dispatch task 5f59eb71-6b76-462c-a639-3013662f2c6e is finished
2021-05-03 22:58:55,103 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(82a197ab-85b8-40a7-9443-4dd7ddd25c27, queue=awx_private_queue)
[2021-05-03 22:58:55,103: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(82a197ab-85b8-40a7-9443-4dd7ddd25c27, queue=awx_private_queue)
2021-05-03 22:58:55,139 DEBUG awx.main.dispatch delivered 82a197ab-85b8-40a7-9443-4dd7ddd25c27 to worker[183] qsize 0
2021-05-03 22:58:55,147 DEBUG awx.main.dispatch task 82a197ab-85b8-40a7-9443-4dd7ddd25c27 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:58:55,163 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:58:55,167 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:58:25.128275+00:00
2021-05-03 22:58:55,189 DEBUG awx.main.dispatch task 82a197ab-85b8-40a7-9443-4dd7ddd25c27 is finished
2021-05-03 22:58:55,397 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(eea7f40d-b970-4966-96c4-a6b4a715dcb3, queue=awx_private_queue)
[2021-05-03 22:58:55,397: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(eea7f40d-b970-4966-96c4-a6b4a715dcb3, queue=awx_private_queue)
2021-05-03 22:58:55,669 DEBUG awx.main.dispatch delivered eea7f40d-b970-4966-96c4-a6b4a715dcb3 to worker[182] qsize 0
2021-05-03 22:58:55,675 DEBUG awx.main.dispatch task eea7f40d-b970-4966-96c4-a6b4a715dcb3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:58:55,685 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:58:55,712 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:58:55,755 DEBUG awx.main.dispatch task eea7f40d-b970-4966-96c4-a6b4a715dcb3 is finished
RESULT 2
OKREADY
2021-05-03 22:59:15,408 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6876aa98-8dbc-4299-afc9-f8bc99e9f762, queue=awx_private_queue)
[2021-05-03 22:59:15,408: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6876aa98-8dbc-4299-afc9-f8bc99e9f762, queue=awx_private_queue)
2021-05-03 22:59:15,436 DEBUG awx.main.dispatch delivered 6876aa98-8dbc-4299-afc9-f8bc99e9f762 to worker[182] qsize 0
2021-05-03 22:59:15,439 DEBUG awx.main.dispatch task 6876aa98-8dbc-4299-afc9-f8bc99e9f762 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:59:15,441 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:59:15,457 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:59:15,487 DEBUG awx.main.dispatch task 6876aa98-8dbc-4299-afc9-f8bc99e9f762 is finished
2021-05-03 22:59:25,102 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(92965a98-1fd7-442a-ad9b-cb656cdda836, queue=awx_private_queue)
[2021-05-03 22:59:25,102: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(92965a98-1fd7-442a-ad9b-cb656cdda836, queue=awx_private_queue)
2021-05-03 22:59:25,118 DEBUG awx.main.dispatch delivered 92965a98-1fd7-442a-ad9b-cb656cdda836 to worker[183] qsize 0
2021-05-03 22:59:25,120 DEBUG awx.main.dispatch task 92965a98-1fd7-442a-ad9b-cb656cdda836 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:59:25,133 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:59:25,136 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:58:55.163223+00:00
2021-05-03 22:59:25,153 DEBUG awx.main.dispatch task 92965a98-1fd7-442a-ad9b-cb656cdda836 is finished
2021-05-03 22:59:35,414 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6c2df2fb-4e84-4868-940e-ac7872c4b7e9, queue=awx_private_queue)
[2021-05-03 22:59:35,414: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6c2df2fb-4e84-4868-940e-ac7872c4b7e9, queue=awx_private_queue)
2021-05-03 22:59:35,439 DEBUG awx.main.dispatch delivered 6c2df2fb-4e84-4868-940e-ac7872c4b7e9 to worker[182] qsize 0
2021-05-03 22:59:35,446 DEBUG awx.main.dispatch task 6c2df2fb-4e84-4868-940e-ac7872c4b7e9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:59:35,450 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:59:35,486 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:59:35,516 DEBUG awx.main.dispatch task 6c2df2fb-4e84-4868-940e-ac7872c4b7e9 is finished
2021-05-03 22:59:54,561 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b9162a79-71f2-460d-8ad1-fdd96d719f43, queue=dev-awx-0)
[2021-05-03 22:59:54,561: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b9162a79-71f2-460d-8ad1-fdd96d719f43, queue=dev-awx-0)
2021-05-03 22:59:54,604 DEBUG awx.main.dispatch delivered b9162a79-71f2-460d-8ad1-fdd96d719f43 to worker[183] qsize 0
2021-05-03 22:59:54,609 DEBUG awx.main.dispatch task b9162a79-71f2-460d-8ad1-fdd96d719f43 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 22:59:54,616 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 22:59:54,656 DEBUG awx.main.dispatch task b9162a79-71f2-460d-8ad1-fdd96d719f43 is finished
2021-05-03 22:59:54,849 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(6b4d36d3-2411-4e73-a565-5b1df9e1a0bc, queue=dev-awx-0)
[2021-05-03 22:59:54,849: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(6b4d36d3-2411-4e73-a565-5b1df9e1a0bc, queue=dev-awx-0)
2021-05-03 22:59:54,875 DEBUG awx.main.dispatch delivered 6b4d36d3-2411-4e73-a565-5b1df9e1a0bc to worker[184] qsize 0
2021-05-03 22:59:54,880 DEBUG awx.main.dispatch task 6b4d36d3-2411-4e73-a565-5b1df9e1a0bc starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 22:59:54,898 DEBUG awx.main.dispatch task 6b4d36d3-2411-4e73-a565-5b1df9e1a0bc is finished
2021-05-03 22:59:55,102 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(63707f68-3508-40d9-8bca-56b19b61b1e9, queue=awx_private_queue)
[2021-05-03 22:59:55,102: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(63707f68-3508-40d9-8bca-56b19b61b1e9, queue=awx_private_queue)
2021-05-03 22:59:55,135 DEBUG awx.main.dispatch delivered 63707f68-3508-40d9-8bca-56b19b61b1e9 to worker[183] qsize 0
2021-05-03 22:59:55,138 DEBUG awx.main.dispatch task 63707f68-3508-40d9-8bca-56b19b61b1e9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 22:59:55,153 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 22:59:55,156 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:59:25.133205+00:00
2021-05-03 22:59:55,169 DEBUG awx.main.dispatch task 63707f68-3508-40d9-8bca-56b19b61b1e9 is finished
2021-05-03 22:59:55,409 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5b43f881-a7a9-45eb-8241-2f473b468d39, queue=awx_private_queue)
[2021-05-03 22:59:55,409: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5b43f881-a7a9-45eb-8241-2f473b468d39, queue=awx_private_queue)
2021-05-03 22:59:55,640 DEBUG awx.main.dispatch delivered 5b43f881-a7a9-45eb-8241-2f473b468d39 to worker[182] qsize 0
2021-05-03 22:59:55,642 DEBUG awx.main.dispatch task 5b43f881-a7a9-45eb-8241-2f473b468d39 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 22:59:55,643 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 22:59:55,666 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 22:59:55,722 DEBUG awx.main.dispatch task 5b43f881-a7a9-45eb-8241-2f473b468d39 is finished
RESULT 2
OKREADY
2021-05-03 23:00:15,437 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(035e6cc8-a3a0-48fa-a047-f0896486136b, queue=awx_private_queue)
[2021-05-03 23:00:15,437: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(035e6cc8-a3a0-48fa-a047-f0896486136b, queue=awx_private_queue)
2021-05-03 23:00:15,468 DEBUG awx.main.dispatch delivered 035e6cc8-a3a0-48fa-a047-f0896486136b to worker[184] qsize 0
2021-05-03 23:00:15,475 DEBUG awx.main.dispatch task 035e6cc8-a3a0-48fa-a047-f0896486136b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:00:15,482 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:00:15,516 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:00:15,656 DEBUG awx.main.dispatch task 035e6cc8-a3a0-48fa-a047-f0896486136b is finished
2021-05-03 23:00:25,105 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(af32f20c-2cb3-497b-89a7-8d8af64796f8, queue=awx_private_queue)
[2021-05-03 23:00:25,105: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(af32f20c-2cb3-497b-89a7-8d8af64796f8, queue=awx_private_queue)
2021-05-03 23:00:25,126 DEBUG awx.main.dispatch delivered af32f20c-2cb3-497b-89a7-8d8af64796f8 to worker[182] qsize 0
2021-05-03 23:00:25,131 DEBUG awx.main.dispatch task af32f20c-2cb3-497b-89a7-8d8af64796f8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:00:25,155 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:00:25,161 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 22:59:55.153920+00:00
2021-05-03 23:00:25,191 DEBUG awx.main.dispatch task af32f20c-2cb3-497b-89a7-8d8af64796f8 is finished
2021-05-03 23:00:35,431 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e46a2d9f-f69b-4ca6-8e0a-b5fd4642b483, queue=awx_private_queue)
[2021-05-03 23:00:35,431: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e46a2d9f-f69b-4ca6-8e0a-b5fd4642b483, queue=awx_private_queue)
2021-05-03 23:00:35,446 DEBUG awx.main.dispatch delivered e46a2d9f-f69b-4ca6-8e0a-b5fd4642b483 to worker[182] qsize 0
2021-05-03 23:00:35,450 DEBUG awx.main.dispatch task e46a2d9f-f69b-4ca6-8e0a-b5fd4642b483 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:00:35,451 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:00:35,498 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:00:35,535 DEBUG awx.main.dispatch task e46a2d9f-f69b-4ca6-8e0a-b5fd4642b483 is finished
2021-05-03 23:00:54,564 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(722725d9-241b-4ddd-9c66-1f5d459c0903, queue=dev-awx-0)
[2021-05-03 23:00:54,564: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(722725d9-241b-4ddd-9c66-1f5d459c0903, queue=dev-awx-0)
2021-05-03 23:00:54,670 DEBUG awx.main.dispatch delivered 722725d9-241b-4ddd-9c66-1f5d459c0903 to worker[184] qsize 0
2021-05-03 23:00:54,672 DEBUG awx.main.dispatch task 722725d9-241b-4ddd-9c66-1f5d459c0903 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:00:54,675 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:00:54,722 DEBUG awx.main.dispatch task 722725d9-241b-4ddd-9c66-1f5d459c0903 is finished
2021-05-03 23:00:54,848 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(7b896a41-94ab-41ec-ab37-5e478bd179f3, queue=dev-awx-0)
[2021-05-03 23:00:54,848: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(7b896a41-94ab-41ec-ab37-5e478bd179f3, queue=dev-awx-0)
2021-05-03 23:00:54,886 DEBUG awx.main.dispatch delivered 7b896a41-94ab-41ec-ab37-5e478bd179f3 to worker[181] qsize 0
2021-05-03 23:00:54,891 DEBUG awx.main.dispatch task 7b896a41-94ab-41ec-ab37-5e478bd179f3 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:00:54,911 DEBUG awx.main.dispatch task 7b896a41-94ab-41ec-ab37-5e478bd179f3 is finished
2021-05-03 23:00:55,102 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f2650386-68a4-4af8-81f8-384c66bec7f1, queue=awx_private_queue)
[2021-05-03 23:00:55,102: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f2650386-68a4-4af8-81f8-384c66bec7f1, queue=awx_private_queue)
2021-05-03 23:00:55,147 DEBUG awx.main.dispatch delivered f2650386-68a4-4af8-81f8-384c66bec7f1 to worker[182] qsize 0
2021-05-03 23:00:55,149 DEBUG awx.main.dispatch task f2650386-68a4-4af8-81f8-384c66bec7f1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:00:55,166 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:00:55,170 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:00:25.156140+00:00
2021-05-03 23:00:55,353 DEBUG awx.main.dispatch task f2650386-68a4-4af8-81f8-384c66bec7f1 is finished
2021-05-03 23:00:55,432 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2725e35a-b711-440d-b12b-a892801676d6, queue=awx_private_queue)
[2021-05-03 23:00:55,432: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2725e35a-b711-440d-b12b-a892801676d6, queue=awx_private_queue)
2021-05-03 23:00:55,624 DEBUG awx.main.dispatch delivered 2725e35a-b711-440d-b12b-a892801676d6 to worker[184] qsize 0
2021-05-03 23:00:55,626 DEBUG awx.main.dispatch task 2725e35a-b711-440d-b12b-a892801676d6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:00:55,629 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:00:55,663 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:00:55,800 DEBUG awx.main.dispatch task 2725e35a-b711-440d-b12b-a892801676d6 is finished
RESULT 2
OKREADY
2021-05-03 23:01:15,446 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cc4dfe9e-ea00-4b4e-afa4-7e3ea0afa44d, queue=awx_private_queue)
[2021-05-03 23:01:15,446: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cc4dfe9e-ea00-4b4e-afa4-7e3ea0afa44d, queue=awx_private_queue)
2021-05-03 23:01:15,460 DEBUG awx.main.dispatch delivered cc4dfe9e-ea00-4b4e-afa4-7e3ea0afa44d to worker[183] qsize 0
2021-05-03 23:01:15,463 DEBUG awx.main.dispatch task cc4dfe9e-ea00-4b4e-afa4-7e3ea0afa44d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:01:15,467 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:01:15,488 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:01:15,540 DEBUG awx.main.dispatch task cc4dfe9e-ea00-4b4e-afa4-7e3ea0afa44d is finished
2021-05-03 23:01:25,102 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(138c6deb-2443-4b76-a2c6-f1cc66e5dfd8, queue=awx_private_queue)
[2021-05-03 23:01:25,102: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(138c6deb-2443-4b76-a2c6-f1cc66e5dfd8, queue=awx_private_queue)
2021-05-03 23:01:25,118 DEBUG awx.main.dispatch delivered 138c6deb-2443-4b76-a2c6-f1cc66e5dfd8 to worker[182] qsize 0
2021-05-03 23:01:25,120 DEBUG awx.main.dispatch task 138c6deb-2443-4b76-a2c6-f1cc66e5dfd8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:01:25,143 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:01:25,149 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:00:55.166360+00:00
2021-05-03 23:01:25,181 DEBUG awx.main.dispatch task 138c6deb-2443-4b76-a2c6-f1cc66e5dfd8 is finished
2021-05-03 23:01:35,453 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(89590787-0562-4c81-8451-7b9be30f99d1, queue=awx_private_queue)
[2021-05-03 23:01:35,453: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(89590787-0562-4c81-8451-7b9be30f99d1, queue=awx_private_queue)
2021-05-03 23:01:35,483 DEBUG awx.main.dispatch delivered 89590787-0562-4c81-8451-7b9be30f99d1 to worker[183] qsize 0
2021-05-03 23:01:35,490 DEBUG awx.main.dispatch task 89590787-0562-4c81-8451-7b9be30f99d1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:01:35,493 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:01:35,587 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:01:35,662 DEBUG awx.main.dispatch task 89590787-0562-4c81-8451-7b9be30f99d1 is finished
2021-05-03 23:01:54,572 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(878a71a2-b4a0-45cb-9c8c-db608252a7be, queue=dev-awx-0)
[2021-05-03 23:01:54,572: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(878a71a2-b4a0-45cb-9c8c-db608252a7be, queue=dev-awx-0)
2021-05-03 23:01:54,609 DEBUG awx.main.dispatch delivered 878a71a2-b4a0-45cb-9c8c-db608252a7be to worker[183] qsize 0
2021-05-03 23:01:54,611 DEBUG awx.main.dispatch task 878a71a2-b4a0-45cb-9c8c-db608252a7be starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:01:54,613 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:01:54,645 DEBUG awx.main.dispatch task 878a71a2-b4a0-45cb-9c8c-db608252a7be is finished
2021-05-03 23:01:54,848 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(72d3865d-fc85-4c83-9636-d2d43392be4b, queue=dev-awx-0)
[2021-05-03 23:01:54,848: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(72d3865d-fc85-4c83-9636-d2d43392be4b, queue=dev-awx-0)
2021-05-03 23:01:54,872 DEBUG awx.main.dispatch delivered 72d3865d-fc85-4c83-9636-d2d43392be4b to worker[183] qsize 0
2021-05-03 23:01:54,884 DEBUG awx.main.dispatch task 72d3865d-fc85-4c83-9636-d2d43392be4b starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:01:54,940 DEBUG awx.main.dispatch task 72d3865d-fc85-4c83-9636-d2d43392be4b is finished
2021-05-03 23:01:55,103 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(de48e499-567d-47a9-8c9e-7c825fc49e11, queue=awx_private_queue)
[2021-05-03 23:01:55,103: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(de48e499-567d-47a9-8c9e-7c825fc49e11, queue=awx_private_queue)
2021-05-03 23:01:55,117 DEBUG awx.main.dispatch delivered de48e499-567d-47a9-8c9e-7c825fc49e11 to worker[184] qsize 0
2021-05-03 23:01:55,120 DEBUG awx.main.dispatch task de48e499-567d-47a9-8c9e-7c825fc49e11 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:01:55,165 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:01:55,171 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:01:25.143526+00:00
2021-05-03 23:01:55,211 DEBUG awx.main.dispatch task de48e499-567d-47a9-8c9e-7c825fc49e11 is finished
2021-05-03 23:01:55,449 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0ffb1712-260f-4510-bdd4-94341b752a0a, queue=awx_private_queue)
[2021-05-03 23:01:55,449: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0ffb1712-260f-4510-bdd4-94341b752a0a, queue=awx_private_queue)
2021-05-03 23:01:55,673 DEBUG awx.main.dispatch delivered 0ffb1712-260f-4510-bdd4-94341b752a0a to worker[184] qsize 0
2021-05-03 23:01:55,676 DEBUG awx.main.dispatch task 0ffb1712-260f-4510-bdd4-94341b752a0a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:01:55,678 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:01:55,748 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:01:55,808 DEBUG awx.main.dispatch task 0ffb1712-260f-4510-bdd4-94341b752a0a is finished
RESULT 2
OKREADY
2021-05-03 23:02:15,456 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(35a619b8-d2a4-4ce1-accf-69f51cefa4bb, queue=awx_private_queue)
[2021-05-03 23:02:15,456: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(35a619b8-d2a4-4ce1-accf-69f51cefa4bb, queue=awx_private_queue)
2021-05-03 23:02:15,478 DEBUG awx.main.dispatch delivered 35a619b8-d2a4-4ce1-accf-69f51cefa4bb to worker[183] qsize 0
2021-05-03 23:02:15,480 DEBUG awx.main.dispatch task 35a619b8-d2a4-4ce1-accf-69f51cefa4bb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:02:15,483 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:02:15,500 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:02:15,544 DEBUG awx.main.dispatch task 35a619b8-d2a4-4ce1-accf-69f51cefa4bb is finished
2021-05-03 23:02:25,103 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3697eb04-ecc2-4a2a-b1df-af7a7f3cdbde, queue=awx_private_queue)
[2021-05-03 23:02:25,103: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3697eb04-ecc2-4a2a-b1df-af7a7f3cdbde, queue=awx_private_queue)
2021-05-03 23:02:25,117 DEBUG awx.main.dispatch delivered 3697eb04-ecc2-4a2a-b1df-af7a7f3cdbde to worker[182] qsize 0
2021-05-03 23:02:25,120 DEBUG awx.main.dispatch task 3697eb04-ecc2-4a2a-b1df-af7a7f3cdbde starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:02:25,145 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:02:25,162 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:01:55.165366+00:00
2021-05-03 23:02:25,180 DEBUG awx.main.dispatch task 3697eb04-ecc2-4a2a-b1df-af7a7f3cdbde is finished
2021-05-03 23:02:35,455 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9cccf92d-8f57-4f06-b911-083c527ea712, queue=awx_private_queue)
[2021-05-03 23:02:35,455: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9cccf92d-8f57-4f06-b911-083c527ea712, queue=awx_private_queue)
2021-05-03 23:02:35,471 DEBUG awx.main.dispatch delivered 9cccf92d-8f57-4f06-b911-083c527ea712 to worker[182] qsize 0
2021-05-03 23:02:35,473 DEBUG awx.main.dispatch task 9cccf92d-8f57-4f06-b911-083c527ea712 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:02:35,474 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:02:35,513 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:02:35,570 DEBUG awx.main.dispatch task 9cccf92d-8f57-4f06-b911-083c527ea712 is finished
2021-05-03 23:02:54,580 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(1ca9a7e3-e5fb-4d79-9909-10f4d724d4e4, queue=dev-awx-0)
[2021-05-03 23:02:54,580: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(1ca9a7e3-e5fb-4d79-9909-10f4d724d4e4, queue=dev-awx-0)
2021-05-03 23:02:54,606 DEBUG awx.main.dispatch delivered 1ca9a7e3-e5fb-4d79-9909-10f4d724d4e4 to worker[184] qsize 0
2021-05-03 23:02:54,609 DEBUG awx.main.dispatch task 1ca9a7e3-e5fb-4d79-9909-10f4d724d4e4 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:02:54,611 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:02:54,638 DEBUG awx.main.dispatch task 1ca9a7e3-e5fb-4d79-9909-10f4d724d4e4 is finished
2021-05-03 23:02:54,848 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f050989b-74da-4c27-902e-03303e823baa, queue=dev-awx-0)
[2021-05-03 23:02:54,848: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f050989b-74da-4c27-902e-03303e823baa, queue=dev-awx-0)
2021-05-03 23:02:54,878 DEBUG awx.main.dispatch delivered f050989b-74da-4c27-902e-03303e823baa to worker[181] qsize 0
2021-05-03 23:02:54,884 DEBUG awx.main.dispatch task f050989b-74da-4c27-902e-03303e823baa starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:02:54,909 DEBUG awx.main.dispatch task f050989b-74da-4c27-902e-03303e823baa is finished
2021-05-03 23:02:55,102 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7c029246-8f3c-4537-804b-a1fb87d9da27, queue=awx_private_queue)
[2021-05-03 23:02:55,102: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7c029246-8f3c-4537-804b-a1fb87d9da27, queue=awx_private_queue)
2021-05-03 23:02:55,116 DEBUG awx.main.dispatch delivered 7c029246-8f3c-4537-804b-a1fb87d9da27 to worker[181] qsize 0
2021-05-03 23:02:55,118 DEBUG awx.main.dispatch task 7c029246-8f3c-4537-804b-a1fb87d9da27 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:02:55,125 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:02:55,128 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:02:25.146035+00:00
2021-05-03 23:02:55,147 DEBUG awx.main.dispatch task 7c029246-8f3c-4537-804b-a1fb87d9da27 is finished
2021-05-03 23:02:55,456 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6f22d420-5708-49bf-886e-bab5b1d1b686, queue=awx_private_queue)
[2021-05-03 23:02:55,456: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6f22d420-5708-49bf-886e-bab5b1d1b686, queue=awx_private_queue)
2021-05-03 23:02:55,670 DEBUG awx.main.dispatch delivered 6f22d420-5708-49bf-886e-bab5b1d1b686 to worker[182] qsize 0
2021-05-03 23:02:55,673 DEBUG awx.main.dispatch task 6f22d420-5708-49bf-886e-bab5b1d1b686 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:02:55,674 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:02:55,691 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:02:55,718 DEBUG awx.main.dispatch task 6f22d420-5708-49bf-886e-bab5b1d1b686 is finished
RESULT 2
OKREADY
2021-05-03 23:03:15,470 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a7a9f112-b600-45e1-b92f-b8143728620d, queue=awx_private_queue)
[2021-05-03 23:03:15,470: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a7a9f112-b600-45e1-b92f-b8143728620d, queue=awx_private_queue)
2021-05-03 23:03:15,503 DEBUG awx.main.dispatch delivered a7a9f112-b600-45e1-b92f-b8143728620d to worker[183] qsize 0
2021-05-03 23:03:15,510 DEBUG awx.main.dispatch task a7a9f112-b600-45e1-b92f-b8143728620d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:03:15,515 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:03:15,563 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:03:15,606 DEBUG awx.main.dispatch task a7a9f112-b600-45e1-b92f-b8143728620d is finished
2021-05-03 23:03:25,106 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c9a7c752-bdde-4600-a67d-25d77ade179b, queue=awx_private_queue)
[2021-05-03 23:03:25,106: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c9a7c752-bdde-4600-a67d-25d77ade179b, queue=awx_private_queue)
2021-05-03 23:03:25,129 DEBUG awx.main.dispatch delivered c9a7c752-bdde-4600-a67d-25d77ade179b to worker[184] qsize 0
2021-05-03 23:03:25,140 DEBUG awx.main.dispatch task c9a7c752-bdde-4600-a67d-25d77ade179b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:03:25,169 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:03:25,178 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:02:55.125851+00:00
2021-05-03 23:03:25,195 DEBUG awx.main.dispatch task c9a7c752-bdde-4600-a67d-25d77ade179b is finished
2021-05-03 23:03:35,466 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5c048933-1ec9-4d5a-a156-22a7901f2760, queue=awx_private_queue)
[2021-05-03 23:03:35,466: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5c048933-1ec9-4d5a-a156-22a7901f2760, queue=awx_private_queue)
2021-05-03 23:03:35,485 DEBUG awx.main.dispatch delivered 5c048933-1ec9-4d5a-a156-22a7901f2760 to worker[182] qsize 0
2021-05-03 23:03:35,492 DEBUG awx.main.dispatch task 5c048933-1ec9-4d5a-a156-22a7901f2760 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:03:35,500 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:03:35,526 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:03:35,555 DEBUG awx.main.dispatch task 5c048933-1ec9-4d5a-a156-22a7901f2760 is finished
2021-05-03 23:03:54,581 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(774d088a-47fc-40a8-8519-6b5ce7fc6f72, queue=dev-awx-0)
[2021-05-03 23:03:54,581: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(774d088a-47fc-40a8-8519-6b5ce7fc6f72, queue=dev-awx-0)
2021-05-03 23:03:54,625 DEBUG awx.main.dispatch delivered 774d088a-47fc-40a8-8519-6b5ce7fc6f72 to worker[181] qsize 0
2021-05-03 23:03:54,632 DEBUG awx.main.dispatch task 774d088a-47fc-40a8-8519-6b5ce7fc6f72 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:03:54,639 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:03:54,679 DEBUG awx.main.dispatch task 774d088a-47fc-40a8-8519-6b5ce7fc6f72 is finished
2021-05-03 23:03:54,848 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(24dbde08-249d-4f12-9a8a-6e0bfd07804d, queue=dev-awx-0)
[2021-05-03 23:03:54,848: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(24dbde08-249d-4f12-9a8a-6e0bfd07804d, queue=dev-awx-0)
2021-05-03 23:03:54,867 DEBUG awx.main.dispatch delivered 24dbde08-249d-4f12-9a8a-6e0bfd07804d to worker[181] qsize 0
2021-05-03 23:03:54,869 DEBUG awx.main.dispatch task 24dbde08-249d-4f12-9a8a-6e0bfd07804d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:03:54,882 DEBUG awx.main.dispatch task 24dbde08-249d-4f12-9a8a-6e0bfd07804d is finished
2021-05-03 23:03:55,103 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b2e14180-2b8e-40c7-a899-909ac1926e54, queue=awx_private_queue)
[2021-05-03 23:03:55,103: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b2e14180-2b8e-40c7-a899-909ac1926e54, queue=awx_private_queue)
2021-05-03 23:03:55,122 DEBUG awx.main.dispatch delivered b2e14180-2b8e-40c7-a899-909ac1926e54 to worker[182] qsize 0
2021-05-03 23:03:55,124 DEBUG awx.main.dispatch task b2e14180-2b8e-40c7-a899-909ac1926e54 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:03:55,134 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:03:55,140 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:03:25.170531+00:00
2021-05-03 23:03:55,179 DEBUG awx.main.dispatch task b2e14180-2b8e-40c7-a899-909ac1926e54 is finished
2021-05-03 23:03:55,465 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(39576a4f-b5b2-4801-a092-ac82bd4786d1, queue=awx_private_queue)
[2021-05-03 23:03:55,465: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(39576a4f-b5b2-4801-a092-ac82bd4786d1, queue=awx_private_queue)
2021-05-03 23:03:55,703 DEBUG awx.main.dispatch delivered 39576a4f-b5b2-4801-a092-ac82bd4786d1 to worker[183] qsize 0
2021-05-03 23:03:55,708 DEBUG awx.main.dispatch task 39576a4f-b5b2-4801-a092-ac82bd4786d1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:03:55,713 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:03:55,745 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:03:55,788 DEBUG awx.main.dispatch task 39576a4f-b5b2-4801-a092-ac82bd4786d1 is finished
RESULT 2
OKREADY
2021-05-03 23:04:15,479 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ff5ad5cb-1466-4a4a-a7fd-cc3e28e95b01, queue=awx_private_queue)
[2021-05-03 23:04:15,479: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ff5ad5cb-1466-4a4a-a7fd-cc3e28e95b01, queue=awx_private_queue)
2021-05-03 23:04:15,494 DEBUG awx.main.dispatch delivered ff5ad5cb-1466-4a4a-a7fd-cc3e28e95b01 to worker[182] qsize 0
2021-05-03 23:04:15,497 DEBUG awx.main.dispatch task ff5ad5cb-1466-4a4a-a7fd-cc3e28e95b01 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:04:15,499 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:04:15,543 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:04:15,683 DEBUG awx.main.dispatch task ff5ad5cb-1466-4a4a-a7fd-cc3e28e95b01 is finished
2021-05-03 23:04:25,103 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(529beb99-e7a5-4b8c-9b6d-652bd5828808, queue=awx_private_queue)
[2021-05-03 23:04:25,103: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(529beb99-e7a5-4b8c-9b6d-652bd5828808, queue=awx_private_queue)
2021-05-03 23:04:25,135 DEBUG awx.main.dispatch delivered 529beb99-e7a5-4b8c-9b6d-652bd5828808 to worker[184] qsize 0
2021-05-03 23:04:25,138 DEBUG awx.main.dispatch task 529beb99-e7a5-4b8c-9b6d-652bd5828808 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:04:25,152 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:04:25,159 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:03:55.135206+00:00
2021-05-03 23:04:25,200 DEBUG awx.main.dispatch task 529beb99-e7a5-4b8c-9b6d-652bd5828808 is finished
2021-05-03 23:04:35,483 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2de26d0b-63b1-4a69-91b8-da98c457436f, queue=awx_private_queue)
[2021-05-03 23:04:35,483: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2de26d0b-63b1-4a69-91b8-da98c457436f, queue=awx_private_queue)
2021-05-03 23:04:35,503 DEBUG awx.main.dispatch delivered 2de26d0b-63b1-4a69-91b8-da98c457436f to worker[182] qsize 0
2021-05-03 23:04:35,510 DEBUG awx.main.dispatch task 2de26d0b-63b1-4a69-91b8-da98c457436f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:04:35,514 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:04:35,552 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:04:35,612 DEBUG awx.main.dispatch task 2de26d0b-63b1-4a69-91b8-da98c457436f is finished
2021-05-03 23:04:54,593 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(88ce9123-3073-4fe9-bb3f-aa320f2ce039, queue=dev-awx-0)
[2021-05-03 23:04:54,593: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(88ce9123-3073-4fe9-bb3f-aa320f2ce039, queue=dev-awx-0)
2021-05-03 23:04:54,781 DEBUG awx.main.dispatch delivered 88ce9123-3073-4fe9-bb3f-aa320f2ce039 to worker[184] qsize 0
2021-05-03 23:04:54,786 DEBUG awx.main.dispatch task 88ce9123-3073-4fe9-bb3f-aa320f2ce039 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:04:54,789 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:04:54,848 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(12a003b1-5ebe-4f53-99ea-fddea2fe6285, queue=dev-awx-0)
[2021-05-03 23:04:54,848: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(12a003b1-5ebe-4f53-99ea-fddea2fe6285, queue=dev-awx-0)
2021-05-03 23:04:54,867 DEBUG awx.main.dispatch delivered 12a003b1-5ebe-4f53-99ea-fddea2fe6285 to worker[181] qsize 0
2021-05-03 23:04:54,872 DEBUG awx.main.dispatch task 12a003b1-5ebe-4f53-99ea-fddea2fe6285 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:04:54,894 DEBUG awx.main.dispatch task 12a003b1-5ebe-4f53-99ea-fddea2fe6285 is finished
2021-05-03 23:04:54,935 DEBUG awx.main.dispatch task 88ce9123-3073-4fe9-bb3f-aa320f2ce039 is finished
2021-05-03 23:04:55,103 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9c51b4be-1047-4637-90d2-ae099c371cc8, queue=awx_private_queue)
[2021-05-03 23:04:55,103: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9c51b4be-1047-4637-90d2-ae099c371cc8, queue=awx_private_queue)
2021-05-03 23:04:55,121 DEBUG awx.main.dispatch delivered 9c51b4be-1047-4637-90d2-ae099c371cc8 to worker[184] qsize 0
2021-05-03 23:04:55,123 DEBUG awx.main.dispatch task 9c51b4be-1047-4637-90d2-ae099c371cc8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:04:55,427 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:04:55,430 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:04:25.152348+00:00
2021-05-03 23:04:55,587 DEBUG awx.main.dispatch task 9c51b4be-1047-4637-90d2-ae099c371cc8 is finished
2021-05-03 23:04:55,481 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2482e7a6-eb80-4ebd-800a-2b1d659a806a, queue=awx_private_queue)
[2021-05-03 23:04:55,481: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2482e7a6-eb80-4ebd-800a-2b1d659a806a, queue=awx_private_queue)
2021-05-03 23:04:55,701 DEBUG awx.main.dispatch delivered 2482e7a6-eb80-4ebd-800a-2b1d659a806a to worker[181] qsize 0
2021-05-03 23:04:55,703 DEBUG awx.main.dispatch task 2482e7a6-eb80-4ebd-800a-2b1d659a806a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:04:55,704 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:04:55,730 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:04:55,764 DEBUG awx.main.dispatch task 2482e7a6-eb80-4ebd-800a-2b1d659a806a is finished
RESULT 2
OKREADY
2021-05-03 23:05:15,481 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ed6afd4c-2ac9-4d6c-ba77-2ae6e53fb08e, queue=awx_private_queue)
[2021-05-03 23:05:15,481: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ed6afd4c-2ac9-4d6c-ba77-2ae6e53fb08e, queue=awx_private_queue)
2021-05-03 23:05:15,496 DEBUG awx.main.dispatch delivered ed6afd4c-2ac9-4d6c-ba77-2ae6e53fb08e to worker[184] qsize 0
2021-05-03 23:05:15,498 DEBUG awx.main.dispatch task ed6afd4c-2ac9-4d6c-ba77-2ae6e53fb08e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:05:15,499 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:05:15,530 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:05:15,564 DEBUG awx.main.dispatch task ed6afd4c-2ac9-4d6c-ba77-2ae6e53fb08e is finished
2021-05-03 23:05:25,103 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0c0bec40-9dea-4f48-84d8-5bc381c63b30, queue=awx_private_queue)
[2021-05-03 23:05:25,103: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0c0bec40-9dea-4f48-84d8-5bc381c63b30, queue=awx_private_queue)
2021-05-03 23:05:25,118 DEBUG awx.main.dispatch delivered 0c0bec40-9dea-4f48-84d8-5bc381c63b30 to worker[184] qsize 0
2021-05-03 23:05:25,120 DEBUG awx.main.dispatch task 0c0bec40-9dea-4f48-84d8-5bc381c63b30 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:05:26,352 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:05:26,490 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:04:55.428026+00:00
2021-05-03 23:05:26,733 DEBUG awx.main.dispatch task 0c0bec40-9dea-4f48-84d8-5bc381c63b30 is finished
2021-05-03 23:05:35,481 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ce45aabc-9005-4599-9dc2-354594f872f3, queue=awx_private_queue)
[2021-05-03 23:05:35,481: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ce45aabc-9005-4599-9dc2-354594f872f3, queue=awx_private_queue)
2021-05-03 23:05:35,514 DEBUG awx.main.dispatch delivered ce45aabc-9005-4599-9dc2-354594f872f3 to worker[182] qsize 0
2021-05-03 23:05:35,516 DEBUG awx.main.dispatch task ce45aabc-9005-4599-9dc2-354594f872f3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:05:35,519 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:05:35,932 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:05:35,986 DEBUG awx.main.dispatch task ce45aabc-9005-4599-9dc2-354594f872f3 is finished
2021-05-03 23:05:54,484 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(de326760-8d49-481c-b4a5-e60074f7e749, queue=dev-awx-0)
[2021-05-03 23:05:54,484: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(de326760-8d49-481c-b4a5-e60074f7e749, queue=dev-awx-0)
2021-05-03 23:05:54,564 DEBUG awx.main.dispatch delivered de326760-8d49-481c-b4a5-e60074f7e749 to worker[184] qsize 0
2021-05-03 23:05:54,567 DEBUG awx.main.dispatch task de326760-8d49-481c-b4a5-e60074f7e749 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-05-03 23:05:54,570 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-05-03 23:05:54,591 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e56e3db5-97f6-4abe-a366-4379748b1366, queue=dev-awx-0)
[2021-05-03 23:05:54,591: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e56e3db5-97f6-4abe-a366-4379748b1366, queue=dev-awx-0)
2021-05-03 23:05:54,602 DEBUG awx.main.dispatch task de326760-8d49-481c-b4a5-e60074f7e749 is finished
2021-05-03 23:05:54,621 DEBUG awx.main.dispatch delivered e56e3db5-97f6-4abe-a366-4379748b1366 to worker[183] qsize 0
2021-05-03 23:05:54,626 DEBUG awx.main.dispatch task e56e3db5-97f6-4abe-a366-4379748b1366 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:05:54,836 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:05:54,848 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(13ee348b-9d5a-433b-be64-f2cdda18762c, queue=dev-awx-0)
[2021-05-03 23:05:54,848: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(13ee348b-9d5a-433b-be64-f2cdda18762c, queue=dev-awx-0)
2021-05-03 23:05:54,867 DEBUG awx.main.dispatch delivered 13ee348b-9d5a-433b-be64-f2cdda18762c to worker[181] qsize 0
2021-05-03 23:05:54,869 DEBUG awx.main.dispatch task 13ee348b-9d5a-433b-be64-f2cdda18762c starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:05:54,870 DEBUG awx.main.dispatch task e56e3db5-97f6-4abe-a366-4379748b1366 is finished
2021-05-03 23:05:54,884 DEBUG awx.main.dispatch task 13ee348b-9d5a-433b-be64-f2cdda18762c is finished
2021-05-03 23:05:55,103 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(fbeb0337-2a64-4abf-8b15-1cade5774551, queue=awx_private_queue)
[2021-05-03 23:05:55,103: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(fbeb0337-2a64-4abf-8b15-1cade5774551, queue=awx_private_queue)
2021-05-03 23:05:55,117 DEBUG awx.main.dispatch delivered fbeb0337-2a64-4abf-8b15-1cade5774551 to worker[184] qsize 0
2021-05-03 23:05:55,119 DEBUG awx.main.dispatch task fbeb0337-2a64-4abf-8b15-1cade5774551 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:05:55,131 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:05:55,138 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:05:26.352389+00:00
2021-05-03 23:05:55,189 DEBUG awx.main.dispatch task fbeb0337-2a64-4abf-8b15-1cade5774551 is finished
2021-05-03 23:05:55,481 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5bfdb41a-2860-4234-88fd-25dd2ae85a91, queue=awx_private_queue)
[2021-05-03 23:05:55,481: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5bfdb41a-2860-4234-88fd-25dd2ae85a91, queue=awx_private_queue)
2021-05-03 23:05:55,498 DEBUG awx.main.dispatch delivered 5bfdb41a-2860-4234-88fd-25dd2ae85a91 to worker[182] qsize 0
2021-05-03 23:05:55,501 DEBUG awx.main.dispatch task 5bfdb41a-2860-4234-88fd-25dd2ae85a91 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:05:55,503 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:05:55,524 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:05:55,554 DEBUG awx.main.dispatch task 5bfdb41a-2860-4234-88fd-25dd2ae85a91 is finished
RESULT 2
OKREADY
2021-05-03 23:06:15,492 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c780a210-17dd-4bac-8e33-d2b406148517, queue=awx_private_queue)
[2021-05-03 23:06:15,492: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c780a210-17dd-4bac-8e33-d2b406148517, queue=awx_private_queue)
2021-05-03 23:06:15,510 DEBUG awx.main.dispatch delivered c780a210-17dd-4bac-8e33-d2b406148517 to worker[184] qsize 0
2021-05-03 23:06:15,515 DEBUG awx.main.dispatch task c780a210-17dd-4bac-8e33-d2b406148517 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:06:15,517 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:06:15,556 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:06:15,595 DEBUG awx.main.dispatch task c780a210-17dd-4bac-8e33-d2b406148517 is finished
2021-05-03 23:06:25,109 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8290353d-72e7-49ef-ae13-6d39cfa5db12, queue=awx_private_queue)
[2021-05-03 23:06:25,109: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8290353d-72e7-49ef-ae13-6d39cfa5db12, queue=awx_private_queue)
2021-05-03 23:06:25,133 DEBUG awx.main.dispatch delivered 8290353d-72e7-49ef-ae13-6d39cfa5db12 to worker[184] qsize 0
2021-05-03 23:06:25,136 DEBUG awx.main.dispatch task 8290353d-72e7-49ef-ae13-6d39cfa5db12 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:06:25,207 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:06:25,212 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:05:55.131664+00:00
2021-05-03 23:06:25,236 DEBUG awx.main.dispatch task 8290353d-72e7-49ef-ae13-6d39cfa5db12 is finished
2021-05-03 23:06:35,493 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2cf22bd3-9b81-4c8c-8285-cc1018ac65df, queue=awx_private_queue)
[2021-05-03 23:06:35,493: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2cf22bd3-9b81-4c8c-8285-cc1018ac65df, queue=awx_private_queue)
2021-05-03 23:06:35,509 DEBUG awx.main.dispatch delivered 2cf22bd3-9b81-4c8c-8285-cc1018ac65df to worker[182] qsize 0
2021-05-03 23:06:35,514 DEBUG awx.main.dispatch task 2cf22bd3-9b81-4c8c-8285-cc1018ac65df starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:06:35,517 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:06:35,556 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:06:35,594 DEBUG awx.main.dispatch task 2cf22bd3-9b81-4c8c-8285-cc1018ac65df is finished
2021-05-03 23:06:54,599 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a6d303bd-35b8-4e3a-8b69-961f4b061b4a, queue=dev-awx-0)
[2021-05-03 23:06:54,599: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a6d303bd-35b8-4e3a-8b69-961f4b061b4a, queue=dev-awx-0)
2021-05-03 23:06:54,848 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b9661ca2-1aba-450e-9794-043aab83e9c0, queue=dev-awx-0)
[2021-05-03 23:06:54,848: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b9661ca2-1aba-450e-9794-043aab83e9c0, queue=dev-awx-0)
2021-05-03 23:06:54,871 DEBUG awx.main.dispatch delivered a6d303bd-35b8-4e3a-8b69-961f4b061b4a to worker[182] qsize 0
2021-05-03 23:06:55,083 DEBUG awx.main.dispatch delivered b9661ca2-1aba-450e-9794-043aab83e9c0 to worker[183] qsize 0
2021-05-03 23:06:55,083 DEBUG awx.main.dispatch task a6d303bd-35b8-4e3a-8b69-961f4b061b4a starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:06:55,085 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:06:55,087 DEBUG awx.main.dispatch task b9661ca2-1aba-450e-9794-043aab83e9c0 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:06:55,105 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(93c11fb1-1250-4e39-8040-d574cf17acde, queue=awx_private_queue)
[2021-05-03 23:06:55,105: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(93c11fb1-1250-4e39-8040-d574cf17acde, queue=awx_private_queue)
2021-05-03 23:06:55,130 DEBUG awx.main.dispatch delivered 93c11fb1-1250-4e39-8040-d574cf17acde to worker[181] qsize 0
2021-05-03 23:06:55,134 DEBUG awx.main.dispatch task 93c11fb1-1250-4e39-8040-d574cf17acde starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:06:55,492 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3085e4bb-8208-4674-b167-5a6b0179b8cd, queue=awx_private_queue)
[2021-05-03 23:06:55,492: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3085e4bb-8208-4674-b167-5a6b0179b8cd, queue=awx_private_queue)
2021-05-03 23:06:55,507 DEBUG awx.main.dispatch delivered 3085e4bb-8208-4674-b167-5a6b0179b8cd to worker[184] qsize 0
2021-05-03 23:06:55,509 DEBUG awx.main.dispatch task 3085e4bb-8208-4674-b167-5a6b0179b8cd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:06:55,511 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:06:57,190 DEBUG awx.main.dispatch task a6d303bd-35b8-4e3a-8b69-961f4b061b4a is finished
2021-05-03 23:06:57,326 DEBUG awx.main.dispatch task b9661ca2-1aba-450e-9794-043aab83e9c0 is finished
2021-05-03 23:06:59,065 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:06:59,073 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:06:25.207914+00:00
2021-05-03 23:06:59,236 DEBUG awx.main.dispatch task 93c11fb1-1250-4e39-8040-d574cf17acde is finished
2021-05-03 23:06:59,686 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:06:59,730 DEBUG awx.main.dispatch task 3085e4bb-8208-4674-b167-5a6b0179b8cd is finished
RESULT 2
OKREADY
2021-05-03 23:07:15,496 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e0d6444f-5757-46df-9b8f-70f7bd9acdcd, queue=awx_private_queue)
[2021-05-03 23:07:15,496: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e0d6444f-5757-46df-9b8f-70f7bd9acdcd, queue=awx_private_queue)
2021-05-03 23:07:15,512 DEBUG awx.main.dispatch delivered e0d6444f-5757-46df-9b8f-70f7bd9acdcd to worker[181] qsize 0
2021-05-03 23:07:15,515 DEBUG awx.main.dispatch task e0d6444f-5757-46df-9b8f-70f7bd9acdcd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:07:15,516 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:07:15,551 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:07:15,618 DEBUG awx.main.dispatch task e0d6444f-5757-46df-9b8f-70f7bd9acdcd is finished
2021-05-03 23:07:25,105 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6481a531-bf3e-4883-b270-fd3237c8cdc4, queue=awx_private_queue)
[2021-05-03 23:07:25,105: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6481a531-bf3e-4883-b270-fd3237c8cdc4, queue=awx_private_queue)
2021-05-03 23:07:25,121 DEBUG awx.main.dispatch delivered 6481a531-bf3e-4883-b270-fd3237c8cdc4 to worker[183] qsize 0
2021-05-03 23:07:25,124 DEBUG awx.main.dispatch task 6481a531-bf3e-4883-b270-fd3237c8cdc4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:07:25,141 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:07:25,152 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:06:59.066278+00:00
2021-05-03 23:07:25,169 DEBUG awx.main.dispatch task 6481a531-bf3e-4883-b270-fd3237c8cdc4 is finished
2021-05-03 23:07:35,496 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(97e3164b-4b46-4a57-bbe3-dbd4f2843294, queue=awx_private_queue)
[2021-05-03 23:07:35,496: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(97e3164b-4b46-4a57-bbe3-dbd4f2843294, queue=awx_private_queue)
2021-05-03 23:07:35,509 DEBUG awx.main.dispatch delivered 97e3164b-4b46-4a57-bbe3-dbd4f2843294 to worker[181] qsize 0
2021-05-03 23:07:35,512 DEBUG awx.main.dispatch task 97e3164b-4b46-4a57-bbe3-dbd4f2843294 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:07:35,513 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:07:35,532 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:07:35,569 DEBUG awx.main.dispatch task 97e3164b-4b46-4a57-bbe3-dbd4f2843294 is finished
2021-05-03 23:07:54,602 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f6c63406-d71f-42f2-9f57-63aaef7d9721, queue=dev-awx-0)
[2021-05-03 23:07:54,602: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f6c63406-d71f-42f2-9f57-63aaef7d9721, queue=dev-awx-0)
2021-05-03 23:07:54,643 DEBUG awx.main.dispatch delivered f6c63406-d71f-42f2-9f57-63aaef7d9721 to worker[181] qsize 0
2021-05-03 23:07:54,646 DEBUG awx.main.dispatch task f6c63406-d71f-42f2-9f57-63aaef7d9721 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:07:54,648 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:07:54,682 DEBUG awx.main.dispatch task f6c63406-d71f-42f2-9f57-63aaef7d9721 is finished
2021-05-03 23:07:54,848 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(0ac21465-37bc-4032-b43c-36eb378e85f4, queue=dev-awx-0)
[2021-05-03 23:07:54,848: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(0ac21465-37bc-4032-b43c-36eb378e85f4, queue=dev-awx-0)
2021-05-03 23:07:54,863 DEBUG awx.main.dispatch delivered 0ac21465-37bc-4032-b43c-36eb378e85f4 to worker[181] qsize 0
2021-05-03 23:07:54,865 DEBUG awx.main.dispatch task 0ac21465-37bc-4032-b43c-36eb378e85f4 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:07:54,886 DEBUG awx.main.dispatch task 0ac21465-37bc-4032-b43c-36eb378e85f4 is finished
2021-05-03 23:07:55,106 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a6a841a1-547c-4e25-a664-d716329a8527, queue=awx_private_queue)
[2021-05-03 23:07:55,106: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a6a841a1-547c-4e25-a664-d716329a8527, queue=awx_private_queue)
2021-05-03 23:07:55,122 DEBUG awx.main.dispatch delivered a6a841a1-547c-4e25-a664-d716329a8527 to worker[184] qsize 0
2021-05-03 23:07:55,301 DEBUG awx.main.dispatch task a6a841a1-547c-4e25-a664-d716329a8527 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:07:55,496 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8957caee-326a-4ff7-ac8f-6934c6714e5d, queue=awx_private_queue)
[2021-05-03 23:07:55,496: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8957caee-326a-4ff7-ac8f-6934c6714e5d, queue=awx_private_queue)
2021-05-03 23:07:55,514 DEBUG awx.main.dispatch delivered 8957caee-326a-4ff7-ac8f-6934c6714e5d to worker[182] qsize 0
2021-05-03 23:07:55,516 DEBUG awx.main.dispatch task 8957caee-326a-4ff7-ac8f-6934c6714e5d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:07:55,519 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:07:55,584 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:07:55,587 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:07:25.141504+00:00
2021-05-03 23:07:55,591 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:07:55,600 DEBUG awx.main.dispatch task a6a841a1-547c-4e25-a664-d716329a8527 is finished
2021-05-03 23:07:55,620 DEBUG awx.main.dispatch task 8957caee-326a-4ff7-ac8f-6934c6714e5d is finished
RESULT 2
OKREADY
2021-05-03 23:08:15,516 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c5687e1d-a47e-496c-9c17-fac2cc2b687e, queue=awx_private_queue)
[2021-05-03 23:08:15,516: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c5687e1d-a47e-496c-9c17-fac2cc2b687e, queue=awx_private_queue)
2021-05-03 23:08:15,544 DEBUG awx.main.dispatch delivered c5687e1d-a47e-496c-9c17-fac2cc2b687e to worker[184] qsize 0
2021-05-03 23:08:15,552 DEBUG awx.main.dispatch task c5687e1d-a47e-496c-9c17-fac2cc2b687e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:08:15,555 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:08:16,083 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:08:16,180 DEBUG awx.main.dispatch task c5687e1d-a47e-496c-9c17-fac2cc2b687e is finished
2021-05-03 23:08:25,105 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(87cada5c-7107-42ed-ad7b-110e25356b7f, queue=awx_private_queue)
[2021-05-03 23:08:25,105: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(87cada5c-7107-42ed-ad7b-110e25356b7f, queue=awx_private_queue)
2021-05-03 23:08:25,123 DEBUG awx.main.dispatch delivered 87cada5c-7107-42ed-ad7b-110e25356b7f to worker[182] qsize 0
2021-05-03 23:08:25,129 DEBUG awx.main.dispatch task 87cada5c-7107-42ed-ad7b-110e25356b7f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:08:25,260 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:08:25,266 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:07:55.584707+00:00
2021-05-03 23:08:25,323 DEBUG awx.main.dispatch task 87cada5c-7107-42ed-ad7b-110e25356b7f is finished
2021-05-03 23:08:35,511 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(383d1409-499f-4087-b7ea-bd805dd79596, queue=awx_private_queue)
[2021-05-03 23:08:35,511: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(383d1409-499f-4087-b7ea-bd805dd79596, queue=awx_private_queue)
2021-05-03 23:08:35,552 DEBUG awx.main.dispatch delivered 383d1409-499f-4087-b7ea-bd805dd79596 to worker[184] qsize 0
2021-05-03 23:08:35,555 DEBUG awx.main.dispatch task 383d1409-499f-4087-b7ea-bd805dd79596 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:08:35,556 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:08:35,639 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:08:35,678 DEBUG awx.main.dispatch task 383d1409-499f-4087-b7ea-bd805dd79596 is finished
RESULT 2
OKREADY
2021-05-03 23:08:54,611 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a2febbeb-6aa3-4269-8b9d-cf3bb8924067, queue=dev-awx-0)
[2021-05-03 23:08:54,611: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a2febbeb-6aa3-4269-8b9d-cf3bb8924067, queue=dev-awx-0)
2021-05-03 23:09:05,441 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(0f7d7b07-3312-48c9-a02e-3d842714b6a7, queue=dev-awx-0)
[2021-05-03 23:09:05,441: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(0f7d7b07-3312-48c9-a02e-3d842714b6a7, queue=dev-awx-0)
2021-05-03 23:09:05,850 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d2611aa2-1ec9-4948-b876-1f7a2a2acd43, queue=awx_private_queue)
[2021-05-03 23:09:05,850: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d2611aa2-1ec9-4948-b876-1f7a2a2acd43, queue=awx_private_queue)
2021-05-03 23:09:05,867 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(74bed993-4a5a-4de2-b384-7a348e3f5714, queue=awx_private_queue)
[2021-05-03 23:09:05,867: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(74bed993-4a5a-4de2-b384-7a348e3f5714, queue=awx_private_queue)
2021-05-03 23:09:05,606 DEBUG awx.main.dispatch delivered a2febbeb-6aa3-4269-8b9d-cf3bb8924067 to worker[182] qsize 0
2021-05-03 23:09:05,888 DEBUG awx.main.dispatch delivered 0f7d7b07-3312-48c9-a02e-3d842714b6a7 to worker[183] qsize 0
2021-05-03 23:09:05,890 DEBUG awx.main.dispatch delivered d2611aa2-1ec9-4948-b876-1f7a2a2acd43 to worker[181] qsize 0
2021-05-03 23:09:05,893 DEBUG awx.main.dispatch delivered 74bed993-4a5a-4de2-b384-7a348e3f5714 to worker[184] qsize 0
2021-05-03 23:09:05,889 DEBUG awx.main.dispatch task a2febbeb-6aa3-4269-8b9d-cf3bb8924067 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:09:05,897 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:09:05,895 DEBUG awx.main.dispatch task 0f7d7b07-3312-48c9-a02e-3d842714b6a7 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:09:05,897 DEBUG awx.main.dispatch task d2611aa2-1ec9-4948-b876-1f7a2a2acd43 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:09:05,906 DEBUG awx.main.dispatch task 74bed993-4a5a-4de2-b384-7a348e3f5714 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:09:05,915 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:09:06,792 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:09:06,801 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:09:06,820 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:08:25.261060+00:00
2021-05-03 23:09:06,821 DEBUG awx.main.dispatch task 0f7d7b07-3312-48c9-a02e-3d842714b6a7 is finished
2021-05-03 23:09:06,975 DEBUG awx.main.dispatch task a2febbeb-6aa3-4269-8b9d-cf3bb8924067 is finished
2021-05-03 23:09:07,007 DEBUG awx.main.dispatch task 74bed993-4a5a-4de2-b384-7a348e3f5714 is finished
2021-05-03 23:09:07,100 DEBUG awx.main.dispatch task d2611aa2-1ec9-4948-b876-1f7a2a2acd43 is finished
2021-05-03 23:09:25,870 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6d643cfa-36ab-486e-ade1-a425f4999ed1, queue=awx_private_queue)
[2021-05-03 23:09:25,870: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6d643cfa-36ab-486e-ade1-a425f4999ed1, queue=awx_private_queue)
2021-05-03 23:09:25,886 DEBUG awx.main.dispatch delivered 6d643cfa-36ab-486e-ade1-a425f4999ed1 to worker[181] qsize 0
2021-05-03 23:09:25,888 DEBUG awx.main.dispatch task 6d643cfa-36ab-486e-ade1-a425f4999ed1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:09:25,890 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:09:35,850 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(fc022082-1d84-4dc3-8540-42cbf704c16a, queue=awx_private_queue)
[2021-05-03 23:09:35,850: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(fc022082-1d84-4dc3-8540-42cbf704c16a, queue=awx_private_queue)
2021-05-03 23:09:35,866 DEBUG awx.main.dispatch delivered fc022082-1d84-4dc3-8540-42cbf704c16a to worker[184] qsize 0
2021-05-03 23:09:35,869 DEBUG awx.main.dispatch task fc022082-1d84-4dc3-8540-42cbf704c16a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:09:45,874 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7414d1d2-33d6-49b5-bf77-58f60fb4095f, queue=awx_private_queue)
[2021-05-03 23:09:45,874: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7414d1d2-33d6-49b5-bf77-58f60fb4095f, queue=awx_private_queue)
2021-05-03 23:09:45,902 DEBUG awx.main.dispatch delivered 7414d1d2-33d6-49b5-bf77-58f60fb4095f to worker[182] qsize 0
2021-05-03 23:09:45,911 DEBUG awx.main.dispatch task 7414d1d2-33d6-49b5-bf77-58f60fb4095f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:09:45,916 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:09:54,608 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(fdf44e3a-bfb5-4ec3-bd82-c747ba8cd162, queue=dev-awx-0)
[2021-05-03 23:09:54,608: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(fdf44e3a-bfb5-4ec3-bd82-c747ba8cd162, queue=dev-awx-0)
2021-05-03 23:10:00,249 DEBUG awx.main.scheduler Starting Scheduler
RESULT 2
OKREADY
2021-05-03 23:10:05,441 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(4b188ea2-54ea-4c9e-bb20-313d211b6fcd, queue=dev-awx-0)
[2021-05-03 23:10:05,441: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(4b188ea2-54ea-4c9e-bb20-313d211b6fcd, queue=dev-awx-0)
2021-05-03 23:10:11,689 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:10:11,699 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:09:06.802313+00:00
2021-05-03 23:10:11,727 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-03 23:10:11,728 DEBUG awx.main.dispatch task 7414d1d2-33d6-49b5-bf77-58f60fb4095f is finished
2021-05-03 23:10:11,724 DEBUG awx.main.dispatch task fc022082-1d84-4dc3-8540-42cbf704c16a is finished
2021-05-03 23:10:12,195 DEBUG awx.main.dispatch delivered fdf44e3a-bfb5-4ec3-bd82-c747ba8cd162 to worker[184] qsize 0
2021-05-03 23:10:12,199 DEBUG awx.main.dispatch delivered 4b188ea2-54ea-4c9e-bb20-313d211b6fcd to worker[182] qsize 0
2021-05-03 23:10:12,199 DEBUG awx.main.dispatch task fdf44e3a-bfb5-4ec3-bd82-c747ba8cd162 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:10:12,202 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:10:12,201 DEBUG awx.main.dispatch task 4b188ea2-54ea-4c9e-bb20-313d211b6fcd starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:10:05,851 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(172e23c9-c68a-40bf-b815-b669ea9d0cf5, queue=awx_private_queue)
[2021-05-03 23:10:05,851: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(172e23c9-c68a-40bf-b815-b669ea9d0cf5, queue=awx_private_queue)
2021-05-03 23:10:12,913 DEBUG awx.main.dispatch delivered 172e23c9-c68a-40bf-b815-b669ea9d0cf5 to worker[183] qsize 0
2021-05-03 23:10:12,917 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(121e0126-effa-47df-b1f4-123dc12fabf2, queue=awx_private_queue)
[2021-05-03 23:10:12,917: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(121e0126-effa-47df-b1f4-123dc12fabf2, queue=awx_private_queue)
2021-05-03 23:10:12,917 DEBUG awx.main.dispatch task 172e23c9-c68a-40bf-b815-b669ea9d0cf5 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:10:12,947 WARNING awx.main.dispatch scaling up worker pid:193
2021-05-03 23:10:12,953 DEBUG awx.main.dispatch delivered 121e0126-effa-47df-b1f4-123dc12fabf2 to worker[193] qsize 0
2021-05-03 23:10:12,961 DEBUG awx.main.dispatch task 121e0126-effa-47df-b1f4-123dc12fabf2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:10:12,965 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:10:20,777 DEBUG awx.main.dispatch task 6d643cfa-36ab-486e-ade1-a425f4999ed1 is finished
2021-05-03 23:10:21,403 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:10:21,415 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:10:11.690242+00:00
2021-05-03 23:10:22,685 DEBUG awx.main.dispatch task 4b188ea2-54ea-4c9e-bb20-313d211b6fcd is finished
2021-05-03 23:10:32,929 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b5819194-61e6-4162-8275-3f2e88ec8926, queue=awx_private_queue)
[2021-05-03 23:10:32,929: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b5819194-61e6-4162-8275-3f2e88ec8926, queue=awx_private_queue)
2021-05-03 23:10:32,946 DEBUG awx.main.dispatch delivered b5819194-61e6-4162-8275-3f2e88ec8926 to worker[181] qsize 0
2021-05-03 23:10:32,949 DEBUG awx.main.dispatch task b5819194-61e6-4162-8275-3f2e88ec8926 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:10:32,950 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:10:33,829 DEBUG awx.main.dispatch task fdf44e3a-bfb5-4ec3-bd82-c747ba8cd162 is finished
2021-05-03 23:10:35,851 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d5169f3b-0673-47ce-a3f2-6cc4c878f16e, queue=awx_private_queue)
[2021-05-03 23:10:35,851: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d5169f3b-0673-47ce-a3f2-6cc4c878f16e, queue=awx_private_queue)
2021-05-03 23:10:35,866 DEBUG awx.main.dispatch delivered d5169f3b-0673-47ce-a3f2-6cc4c878f16e to worker[184] qsize 0
2021-05-03 23:10:35,869 DEBUG awx.main.dispatch task d5169f3b-0673-47ce-a3f2-6cc4c878f16e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:10:52,938 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5589bf38-b12e-49bf-8403-115a4cf70e9e, queue=awx_private_queue)
[2021-05-03 23:10:52,938: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5589bf38-b12e-49bf-8403-115a4cf70e9e, queue=awx_private_queue)
2021-05-03 23:10:52,958 DEBUG awx.main.dispatch delivered 5589bf38-b12e-49bf-8403-115a4cf70e9e to worker[182] qsize 0
2021-05-03 23:10:52,963 DEBUG awx.main.dispatch task 5589bf38-b12e-49bf-8403-115a4cf70e9e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:10:52,964 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:10:54,609 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(4902215b-84cd-4e07-b991-226b4b591964, queue=dev-awx-0)
[2021-05-03 23:10:54,609: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(4902215b-84cd-4e07-b991-226b4b591964, queue=dev-awx-0)
RESULT 2
OKREADY
2021-05-03 23:11:05,442 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(6b681d24-ed89-4e74-a281-76ac86cfc878, queue=dev-awx-0)
[2021-05-03 23:11:05,442: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(6b681d24-ed89-4e74-a281-76ac86cfc878, queue=dev-awx-0)
2021-05-03 23:11:05,851 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(04e1785b-2d56-4756-bd58-4c3d1325017e, queue=awx_private_queue)
[2021-05-03 23:11:05,851: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(04e1785b-2d56-4756-bd58-4c3d1325017e, queue=awx_private_queue)
2021-05-03 23:11:10,190 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:11:11,827 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-05-03 23:11:11,827 DEBUG awx.main.dispatch task d5169f3b-0673-47ce-a3f2-6cc4c878f16e is finished
2021-05-03 23:11:12,937 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d82ae835-022e-4e2f-9108-3683436ad331, queue=awx_private_queue)
[2021-05-03 23:11:12,937: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d82ae835-022e-4e2f-9108-3683436ad331, queue=awx_private_queue)
2021-05-03 23:11:16,321 DEBUG awx.main.dispatch task 172e23c9-c68a-40bf-b815-b669ea9d0cf5 is finished
2021-05-03 23:11:17,037 DEBUG awx.main.dispatch delivered 4902215b-84cd-4e07-b991-226b4b591964 to worker[184] qsize 0
2021-05-03 23:11:17,041 DEBUG awx.main.dispatch task 4902215b-84cd-4e07-b991-226b4b591964 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:11:17,042 DEBUG awx.main.dispatch delivered 6b681d24-ed89-4e74-a281-76ac86cfc878 to worker[183] qsize 0
2021-05-03 23:11:17,043 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:11:17,045 DEBUG awx.main.dispatch task 6b681d24-ed89-4e74-a281-76ac86cfc878 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:11:17,065 WARNING awx.main.dispatch scaling up worker pid:195
2021-05-03 23:11:17,072 DEBUG awx.main.dispatch delivered 04e1785b-2d56-4756-bd58-4c3d1325017e to worker[195] qsize 0
2021-05-03 23:11:17,077 DEBUG awx.main.dispatch task 04e1785b-2d56-4756-bd58-4c3d1325017e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:11:17,085 WARNING awx.main.dispatch scaling up worker pid:197
2021-05-03 23:11:17,091 DEBUG awx.main.dispatch delivered d82ae835-022e-4e2f-9108-3683436ad331 to worker[197] qsize 0
2021-05-03 23:11:17,097 DEBUG awx.main.dispatch task d82ae835-022e-4e2f-9108-3683436ad331 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:11:17,099 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:11:17,141 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-03 23:11:17,141 DEBUG awx.main.dispatch task 5589bf38-b12e-49bf-8403-115a4cf70e9e is finished
2021-05-03 23:11:18,205 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-03 23:11:18,205 DEBUG awx.main.dispatch task b5819194-61e6-4162-8275-3f2e88ec8926 is finished
2021-05-03 23:11:19,097 DEBUG awx.main.dispatch task 121e0126-effa-47df-b1f4-123dc12fabf2 is finished
2021-05-03 23:11:19,183 DEBUG awx.main.dispatch task 6b681d24-ed89-4e74-a281-76ac86cfc878 is finished
2021-05-03 23:11:19,707 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:11:19,729 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:10:21.403401+00:00
2021-05-03 23:11:20,117 DEBUG awx.main.dispatch task 04e1785b-2d56-4756-bd58-4c3d1325017e is finished
2021-05-03 23:11:20,297 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:11:20,310 DEBUG awx.main.dispatch task 4902215b-84cd-4e07-b991-226b4b591964 is finished
2021-05-03 23:11:20,393 DEBUG awx.main.dispatch task d82ae835-022e-4e2f-9108-3683436ad331 is finished
2021-05-03 23:11:32,952 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(43c5230d-307f-4b30-94bf-1cc1ac10c0a9, queue=awx_private_queue)
[2021-05-03 23:11:32,952: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(43c5230d-307f-4b30-94bf-1cc1ac10c0a9, queue=awx_private_queue)
2021-05-03 23:11:32,982 DEBUG awx.main.dispatch delivered 43c5230d-307f-4b30-94bf-1cc1ac10c0a9 to worker[183] qsize 0
2021-05-03 23:11:32,986 DEBUG awx.main.dispatch task 43c5230d-307f-4b30-94bf-1cc1ac10c0a9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:11:32,990 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:11:35,851 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6e655926-8240-4a7d-97fa-8154e38f3d00, queue=awx_private_queue)
[2021-05-03 23:11:35,851: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6e655926-8240-4a7d-97fa-8154e38f3d00, queue=awx_private_queue)
2021-05-03 23:11:35,866 DEBUG awx.main.dispatch delivered 6e655926-8240-4a7d-97fa-8154e38f3d00 to worker[195] qsize 0
2021-05-03 23:11:35,868 DEBUG awx.main.dispatch task 6e655926-8240-4a7d-97fa-8154e38f3d00 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:11:52,950 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ab3b3a70-e1b1-4755-bb1d-8454a61622ca, queue=awx_private_queue)
[2021-05-03 23:11:52,950: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ab3b3a70-e1b1-4755-bb1d-8454a61622ca, queue=awx_private_queue)
2021-05-03 23:11:52,966 DEBUG awx.main.dispatch delivered ab3b3a70-e1b1-4755-bb1d-8454a61622ca to worker[181] qsize 0
2021-05-03 23:11:52,972 DEBUG awx.main.dispatch task ab3b3a70-e1b1-4755-bb1d-8454a61622ca starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:11:52,977 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:11:54,608 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6d735149-ce54-4f84-9738-84d300bea1e3, queue=dev-awx-0)
[2021-05-03 23:11:54,608: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6d735149-ce54-4f84-9738-84d300bea1e3, queue=dev-awx-0)
2021-05-03 23:11:54,623 WARNING awx.main.dispatch scaling down worker pid:182
2021-05-03 23:11:54,626 WARNING awx.main.dispatch scaling down worker pid:184
2021-05-03 23:11:54,628 WARNING awx.main.dispatch scaling down worker pid:193
2021-05-03 23:11:54,627 WARNING awx.main.dispatch worker exiting gracefully pid:182
2021-05-03 23:11:54,629 WARNING awx.main.dispatch worker exiting gracefully pid:184
2021-05-03 23:11:54,646 WARNING awx.main.dispatch worker exiting gracefully pid:193
RESULT 2
OKREADY
2021-05-03 23:12:05,443 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b3f063e5-5e8b-4059-bb96-8652d6a67c5f, queue=dev-awx-0)
[2021-05-03 23:12:05,443: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b3f063e5-5e8b-4059-bb96-8652d6a67c5f, queue=dev-awx-0)
2021-05-03 23:12:05,851 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9af44bb7-1027-470b-ade8-9838b4b32085, queue=awx_private_queue)
[2021-05-03 23:12:05,851: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9af44bb7-1027-470b-ade8-9838b4b32085, queue=awx_private_queue)
2021-05-03 23:12:12,949 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4ba89d01-d34a-4e25-8c28-fd4602ef9b7b, queue=awx_private_queue)
[2021-05-03 23:12:12,949: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4ba89d01-d34a-4e25-8c28-fd4602ef9b7b, queue=awx_private_queue)
2021-05-03 23:12:29,341 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:12:29,459 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:11:19.708256+00:00
2021-05-03 23:12:29,460 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:12:29,613 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-03 23:12:29,613 DEBUG awx.main.dispatch task 43c5230d-307f-4b30-94bf-1cc1ac10c0a9 is finished
2021-05-03 23:12:31,353 DEBUG awx.main.dispatch delivered 6d735149-ce54-4f84-9738-84d300bea1e3 to worker[197] qsize 0
2021-05-03 23:12:31,357 DEBUG awx.main.dispatch delivered b3f063e5-5e8b-4059-bb96-8652d6a67c5f to worker[183] qsize 0
2021-05-03 23:12:31,359 DEBUG awx.main.dispatch task b3f063e5-5e8b-4059-bb96-8652d6a67c5f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:12:31,359 DEBUG awx.main.dispatch task 6d735149-ce54-4f84-9738-84d300bea1e3 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:12:31,364 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:12:31,372 WARNING awx.main.dispatch scaling up worker pid:202
2021-05-03 23:12:31,378 DEBUG awx.main.dispatch delivered 9af44bb7-1027-470b-ade8-9838b4b32085 to worker[202] qsize 0
2021-05-03 23:12:31,382 DEBUG awx.main.dispatch task 9af44bb7-1027-470b-ade8-9838b4b32085 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:12:31,388 WARNING awx.main.dispatch scaling up worker pid:204
2021-05-03 23:12:31,394 DEBUG awx.main.dispatch delivered 4ba89d01-d34a-4e25-8c28-fd4602ef9b7b to worker[204] qsize 0
2021-05-03 23:12:31,399 DEBUG awx.main.dispatch task 4ba89d01-d34a-4e25-8c28-fd4602ef9b7b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:12:31,403 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:12:31,547 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-05-03 23:12:31,547 DEBUG awx.main.dispatch task 9af44bb7-1027-470b-ade8-9838b4b32085 is finished
2021-05-03 23:12:31,570 DEBUG awx.main.dispatch task b3f063e5-5e8b-4059-bb96-8652d6a67c5f is finished
2021-05-03 23:12:31,598 DEBUG awx.main.dispatch task 6d735149-ce54-4f84-9738-84d300bea1e3 is finished
2021-05-03 23:12:32,949 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f6f1dae4-6237-4e42-b1ed-8c89c6b5f4bf, queue=awx_private_queue)
[2021-05-03 23:12:32,949: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f6f1dae4-6237-4e42-b1ed-8c89c6b5f4bf, queue=awx_private_queue)
2021-05-03 23:12:32,965 DEBUG awx.main.dispatch delivered f6f1dae4-6237-4e42-b1ed-8c89c6b5f4bf to worker[202] qsize 0
2021-05-03 23:12:32,969 DEBUG awx.main.dispatch task f6f1dae4-6237-4e42-b1ed-8c89c6b5f4bf starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:12:32,970 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:12:33,782 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-03 23:12:33,782 DEBUG awx.main.dispatch task f6f1dae4-6237-4e42-b1ed-8c89c6b5f4bf is finished
2021-05-03 23:12:33,985 DEBUG awx.main.dispatch task ab3b3a70-e1b1-4755-bb1d-8454a61622ca is finished
2021-05-03 23:12:34,533 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:12:35,851 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(81db915d-86c5-4380-a931-77fadd9aca48, queue=awx_private_queue)
[2021-05-03 23:12:35,851: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(81db915d-86c5-4380-a931-77fadd9aca48, queue=awx_private_queue)
2021-05-03 23:12:35,868 DEBUG awx.main.dispatch delivered 81db915d-86c5-4380-a931-77fadd9aca48 to worker[181] qsize 0
2021-05-03 23:12:35,870 DEBUG awx.main.dispatch task 81db915d-86c5-4380-a931-77fadd9aca48 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:12:35,930 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-05-03 23:12:35,931 DEBUG awx.main.dispatch task 81db915d-86c5-4380-a931-77fadd9aca48 is finished
2021-05-03 23:12:36,037 DEBUG awx.main.dispatch task 4ba89d01-d34a-4e25-8c28-fd4602ef9b7b is finished
2021-05-03 23:12:38,333 DEBUG awx.main.dispatch task 6e655926-8240-4a7d-97fa-8154e38f3d00 is finished
2021-05-03 23:12:52,950 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1954a783-4ad9-40c3-b571-0e540ee08f77, queue=awx_private_queue)
[2021-05-03 23:12:52,950: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1954a783-4ad9-40c3-b571-0e540ee08f77, queue=awx_private_queue)
2021-05-03 23:12:52,967 DEBUG awx.main.dispatch delivered 1954a783-4ad9-40c3-b571-0e540ee08f77 to worker[202] qsize 0
2021-05-03 23:12:52,971 DEBUG awx.main.dispatch task 1954a783-4ad9-40c3-b571-0e540ee08f77 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:12:52,973 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:12:54,608 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e4d89fca-09d4-4de7-a233-b593acfbbe4e, queue=dev-awx-0)
[2021-05-03 23:12:54,608: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e4d89fca-09d4-4de7-a233-b593acfbbe4e, queue=dev-awx-0)
2021-05-03 23:12:54,623 WARNING awx.main.dispatch scaling down worker pid:181
2021-05-03 23:12:54,626 WARNING awx.main.dispatch scaling down worker pid:183
2021-05-03 23:12:54,630 WARNING awx.main.dispatch worker exiting gracefully pid:181
2021-05-03 23:12:54,631 WARNING awx.main.dispatch worker exiting gracefully pid:183
RESULT 2
OKREADY
2021-05-03 23:13:01,753 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:13:02,353 DEBUG awx.main.dispatch task 1954a783-4ad9-40c3-b571-0e540ee08f77 is finished
2021-05-03 23:13:03,079 DEBUG awx.main.dispatch delivered e4d89fca-09d4-4de7-a233-b593acfbbe4e to worker[204] qsize 0
2021-05-03 23:13:03,082 DEBUG awx.main.dispatch task e4d89fca-09d4-4de7-a233-b593acfbbe4e starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:13:03,083 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:13:03,905 DEBUG awx.main.dispatch task e4d89fca-09d4-4de7-a233-b593acfbbe4e is finished
2021-05-03 23:13:05,443 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(879a3bdf-fb90-416f-86da-58bcca18ac53, queue=dev-awx-0)
[2021-05-03 23:13:05,443: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(879a3bdf-fb90-416f-86da-58bcca18ac53, queue=dev-awx-0)
2021-05-03 23:13:05,458 DEBUG awx.main.dispatch delivered 879a3bdf-fb90-416f-86da-58bcca18ac53 to worker[202] qsize 0
2021-05-03 23:13:05,460 DEBUG awx.main.dispatch task 879a3bdf-fb90-416f-86da-58bcca18ac53 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:13:05,504 DEBUG awx.main.dispatch task 879a3bdf-fb90-416f-86da-58bcca18ac53 is finished
2021-05-03 23:13:05,851 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e5093347-9cf7-4c55-a352-38ebbbc3aca9, queue=awx_private_queue)
[2021-05-03 23:13:05,851: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e5093347-9cf7-4c55-a352-38ebbbc3aca9, queue=awx_private_queue)
2021-05-03 23:13:05,867 DEBUG awx.main.dispatch delivered e5093347-9cf7-4c55-a352-38ebbbc3aca9 to worker[204] qsize 0
2021-05-03 23:13:05,870 DEBUG awx.main.dispatch task e5093347-9cf7-4c55-a352-38ebbbc3aca9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:13:05,912 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:13:05,917 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:12:29.341390+00:00
2021-05-03 23:13:05,967 DEBUG awx.main.dispatch task e5093347-9cf7-4c55-a352-38ebbbc3aca9 is finished
2021-05-03 23:13:12,953 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5b138695-4fe1-471c-becd-d42d0e482718, queue=awx_private_queue)
[2021-05-03 23:13:12,953: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5b138695-4fe1-471c-becd-d42d0e482718, queue=awx_private_queue)
2021-05-03 23:13:12,977 DEBUG awx.main.dispatch delivered 5b138695-4fe1-471c-becd-d42d0e482718 to worker[202] qsize 0
2021-05-03 23:13:12,982 DEBUG awx.main.dispatch task 5b138695-4fe1-471c-becd-d42d0e482718 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:13:12,983 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:13:13,016 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:13:13,070 DEBUG awx.main.dispatch task 5b138695-4fe1-471c-becd-d42d0e482718 is finished
2021-05-03 23:13:32,952 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e206ca1a-bd3f-4cf4-9788-271afddd8043, queue=awx_private_queue)
[2021-05-03 23:13:32,952: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e206ca1a-bd3f-4cf4-9788-271afddd8043, queue=awx_private_queue)
2021-05-03 23:13:34,935 DEBUG awx.main.dispatch delivered e206ca1a-bd3f-4cf4-9788-271afddd8043 to worker[204] qsize 0
2021-05-03 23:13:35,299 DEBUG awx.main.dispatch task e206ca1a-bd3f-4cf4-9788-271afddd8043 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:13:35,303 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:13:35,851 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d4659d6d-0463-474f-98c0-a3d1af634d90, queue=awx_private_queue)
[2021-05-03 23:13:35,851: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d4659d6d-0463-474f-98c0-a3d1af634d90, queue=awx_private_queue)
2021-05-03 23:13:35,866 DEBUG awx.main.dispatch delivered d4659d6d-0463-474f-98c0-a3d1af634d90 to worker[202] qsize 0
2021-05-03 23:13:35,870 DEBUG awx.main.dispatch task d4659d6d-0463-474f-98c0-a3d1af634d90 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:13:37,013 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:13:37,018 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:13:05.913091+00:00
2021-05-03 23:13:37,071 DEBUG awx.main.dispatch task d4659d6d-0463-474f-98c0-a3d1af634d90 is finished
2021-05-03 23:13:37,082 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:13:38,308 DEBUG awx.main.dispatch task e206ca1a-bd3f-4cf4-9788-271afddd8043 is finished
2021-05-03 23:13:54,608 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(50bc5445-2a58-489d-8168-73e864db024a, queue=dev-awx-0)
[2021-05-03 23:13:54,608: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(50bc5445-2a58-489d-8168-73e864db024a, queue=dev-awx-0)
2021-05-03 23:13:54,626 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(936810c2-e512-4ae5-bb9b-bf092cb8f9a6, queue=awx_private_queue)
[2021-05-03 23:13:54,626: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(936810c2-e512-4ae5-bb9b-bf092cb8f9a6, queue=awx_private_queue)
2021-05-03 23:13:55,971 DEBUG awx.main.dispatch delivered 50bc5445-2a58-489d-8168-73e864db024a to worker[204] qsize 0
2021-05-03 23:13:55,973 DEBUG awx.main.dispatch delivered 936810c2-e512-4ae5-bb9b-bf092cb8f9a6 to worker[202] qsize 0
2021-05-03 23:13:55,973 DEBUG awx.main.dispatch task 50bc5445-2a58-489d-8168-73e864db024a starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:13:55,975 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:13:55,975 DEBUG awx.main.dispatch task 936810c2-e512-4ae5-bb9b-bf092cb8f9a6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:13:55,977 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:13:57,692 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:13:57,920 DEBUG awx.main.dispatch task 936810c2-e512-4ae5-bb9b-bf092cb8f9a6 is finished
2021-05-03 23:13:58,532 DEBUG awx.main.dispatch task 50bc5445-2a58-489d-8168-73e864db024a is finished
RESULT 2
OKREADY
2021-05-03 23:14:05,443 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(04f8a5f5-6818-469a-89ec-30993d87a382, queue=dev-awx-0)
[2021-05-03 23:14:05,443: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(04f8a5f5-6818-469a-89ec-30993d87a382, queue=dev-awx-0)
2021-05-03 23:14:05,460 DEBUG awx.main.dispatch delivered 04f8a5f5-6818-469a-89ec-30993d87a382 to worker[197] qsize 0
2021-05-03 23:14:05,465 DEBUG awx.main.dispatch task 04f8a5f5-6818-469a-89ec-30993d87a382 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:14:05,494 DEBUG awx.main.dispatch task 04f8a5f5-6818-469a-89ec-30993d87a382 is finished
2021-05-03 23:14:05,851 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d4ace549-9956-4ecd-b02b-b557f035d6e4, queue=awx_private_queue)
[2021-05-03 23:14:05,851: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d4ace549-9956-4ecd-b02b-b557f035d6e4, queue=awx_private_queue)
2021-05-03 23:14:05,878 DEBUG awx.main.dispatch delivered d4ace549-9956-4ecd-b02b-b557f035d6e4 to worker[197] qsize 0
2021-05-03 23:14:05,881 DEBUG awx.main.dispatch task d4ace549-9956-4ecd-b02b-b557f035d6e4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:14:05,895 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:14:05,900 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:13:37.013372+00:00
2021-05-03 23:14:05,921 DEBUG awx.main.dispatch task d4ace549-9956-4ecd-b02b-b557f035d6e4 is finished
2021-05-03 23:14:14,627 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(38dc3be2-71cb-484d-8b59-19d0fb64dbc6, queue=awx_private_queue)
[2021-05-03 23:14:14,627: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(38dc3be2-71cb-484d-8b59-19d0fb64dbc6, queue=awx_private_queue)
2021-05-03 23:14:14,649 DEBUG awx.main.dispatch delivered 38dc3be2-71cb-484d-8b59-19d0fb64dbc6 to worker[204] qsize 0
2021-05-03 23:14:14,652 DEBUG awx.main.dispatch task 38dc3be2-71cb-484d-8b59-19d0fb64dbc6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:14:14,654 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:14:15,038 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:14:15,156 DEBUG awx.main.dispatch task 38dc3be2-71cb-484d-8b59-19d0fb64dbc6 is finished
2021-05-03 23:14:34,634 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f68f81a2-d629-42c4-93a3-2c480e1e5c55, queue=awx_private_queue)
[2021-05-03 23:14:34,634: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f68f81a2-d629-42c4-93a3-2c480e1e5c55, queue=awx_private_queue)
2021-05-03 23:14:34,650 DEBUG awx.main.dispatch delivered f68f81a2-d629-42c4-93a3-2c480e1e5c55 to worker[204] qsize 0
2021-05-03 23:14:34,652 DEBUG awx.main.dispatch task f68f81a2-d629-42c4-93a3-2c480e1e5c55 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:14:34,654 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:14:35,192 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:14:35,247 DEBUG awx.main.dispatch task f68f81a2-d629-42c4-93a3-2c480e1e5c55 is finished
2021-05-03 23:14:35,851 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4b629b2f-2dd0-4007-86d2-fceadd37cdc6, queue=awx_private_queue)
[2021-05-03 23:14:35,851: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4b629b2f-2dd0-4007-86d2-fceadd37cdc6, queue=awx_private_queue)
2021-05-03 23:14:35,868 DEBUG awx.main.dispatch delivered 4b629b2f-2dd0-4007-86d2-fceadd37cdc6 to worker[204] qsize 0
2021-05-03 23:14:36,048 DEBUG awx.main.dispatch task 4b629b2f-2dd0-4007-86d2-fceadd37cdc6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:14:36,076 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:14:36,081 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:14:05.895411+00:00
2021-05-03 23:14:36,172 DEBUG awx.main.dispatch task 4b629b2f-2dd0-4007-86d2-fceadd37cdc6 is finished
2021-05-03 23:14:54,615 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8e3a36d7-ba6c-4785-89da-d812c96cc35c, queue=dev-awx-0)
[2021-05-03 23:14:54,615: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8e3a36d7-ba6c-4785-89da-d812c96cc35c, queue=dev-awx-0)
2021-05-03 23:14:54,645 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(99068515-2833-40c5-b568-1e764dbcc04c, queue=awx_private_queue)
[2021-05-03 23:14:54,645: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(99068515-2833-40c5-b568-1e764dbcc04c, queue=awx_private_queue)
2021-05-03 23:14:54,800 DEBUG awx.main.dispatch delivered 8e3a36d7-ba6c-4785-89da-d812c96cc35c to worker[195] qsize 0
2021-05-03 23:14:54,803 DEBUG awx.main.dispatch delivered 99068515-2833-40c5-b568-1e764dbcc04c to worker[204] qsize 0
2021-05-03 23:14:54,806 DEBUG awx.main.dispatch task 99068515-2833-40c5-b568-1e764dbcc04c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:14:54,808 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:14:54,804 DEBUG awx.main.dispatch task 8e3a36d7-ba6c-4785-89da-d812c96cc35c starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:14:54,810 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:14:55,714 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:14:55,780 DEBUG awx.main.dispatch task 99068515-2833-40c5-b568-1e764dbcc04c is finished
2021-05-03 23:14:56,453 DEBUG awx.main.dispatch task 8e3a36d7-ba6c-4785-89da-d812c96cc35c is finished
RESULT 2
OKREADY
2021-05-03 23:15:05,454 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(561bc01b-bc77-4899-9358-e5ebbfedfd02, queue=dev-awx-0)
[2021-05-03 23:15:05,454: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(561bc01b-bc77-4899-9358-e5ebbfedfd02, queue=dev-awx-0)
2021-05-03 23:15:05,480 DEBUG awx.main.dispatch delivered 561bc01b-bc77-4899-9358-e5ebbfedfd02 to worker[204] qsize 0
2021-05-03 23:15:05,487 DEBUG awx.main.dispatch task 561bc01b-bc77-4899-9358-e5ebbfedfd02 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:15:05,851 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(38f9fce0-e42b-4919-81be-534a14f5beff, queue=awx_private_queue)
[2021-05-03 23:15:05,851: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(38f9fce0-e42b-4919-81be-534a14f5beff, queue=awx_private_queue)
2021-05-03 23:15:05,869 DEBUG awx.main.dispatch delivered 38f9fce0-e42b-4919-81be-534a14f5beff to worker[202] qsize 0
2021-05-03 23:15:05,873 DEBUG awx.main.dispatch task 38f9fce0-e42b-4919-81be-534a14f5beff starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:15:14,645 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4080c059-a74c-4d8b-a71a-80e536551b71, queue=awx_private_queue)
[2021-05-03 23:15:14,645: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4080c059-a74c-4d8b-a71a-80e536551b71, queue=awx_private_queue)
2021-05-03 23:15:14,668 DEBUG awx.main.dispatch delivered 4080c059-a74c-4d8b-a71a-80e536551b71 to worker[195] qsize 0
2021-05-03 23:15:14,673 DEBUG awx.main.dispatch task 4080c059-a74c-4d8b-a71a-80e536551b71 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:15:14,674 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:15:22,734 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:15:22,792 DEBUG awx.main.dispatch task 561bc01b-bc77-4899-9358-e5ebbfedfd02 is finished
2021-05-03 23:15:22,798 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:14:36.077332+00:00
2021-05-03 23:15:22,902 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:15:32,232 DEBUG awx.main.dispatch task 38f9fce0-e42b-4919-81be-534a14f5beff is finished
2021-05-03 23:15:34,654 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3e596d01-b651-43cd-82ed-eaa8b503483d, queue=awx_private_queue)
[2021-05-03 23:15:34,654: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3e596d01-b651-43cd-82ed-eaa8b503483d, queue=awx_private_queue)
2021-05-03 23:15:34,672 DEBUG awx.main.dispatch delivered 3e596d01-b651-43cd-82ed-eaa8b503483d to worker[204] qsize 0
2021-05-03 23:15:34,675 DEBUG awx.main.dispatch task 3e596d01-b651-43cd-82ed-eaa8b503483d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:15:34,676 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:15:35,851 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6f061064-3784-4032-95e0-2bc98f4c0a7b, queue=awx_private_queue)
[2021-05-03 23:15:35,851: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6f061064-3784-4032-95e0-2bc98f4c0a7b, queue=awx_private_queue)
2021-05-03 23:15:35,867 DEBUG awx.main.dispatch delivered 6f061064-3784-4032-95e0-2bc98f4c0a7b to worker[197] qsize 0
2021-05-03 23:15:35,871 DEBUG awx.main.dispatch task 6f061064-3784-4032-95e0-2bc98f4c0a7b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:15:41,693 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-03 23:15:41,693 DEBUG awx.main.dispatch task 4080c059-a74c-4d8b-a71a-80e536551b71 is finished
2021-05-03 23:15:41,693 DEBUG awx.main.dispatch task 3e596d01-b651-43cd-82ed-eaa8b503483d is finished
2021-05-03 23:15:42,054 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:15:42,072 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:15:22.734736+00:00
2021-05-03 23:15:43,074 DEBUG awx.main.dispatch task 6f061064-3784-4032-95e0-2bc98f4c0a7b is finished
2021-05-03 23:15:54,484 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(bfd905d5-e69b-47c7-9ae6-3ec4453a1ca1, queue=dev-awx-0)
[2021-05-03 23:15:54,484: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(bfd905d5-e69b-47c7-9ae6-3ec4453a1ca1, queue=dev-awx-0)
2021-05-03 23:15:54,502 DEBUG awx.main.dispatch delivered bfd905d5-e69b-47c7-9ae6-3ec4453a1ca1 to worker[197] qsize 0
2021-05-03 23:15:54,506 DEBUG awx.main.dispatch task bfd905d5-e69b-47c7-9ae6-3ec4453a1ca1 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-05-03 23:15:54,508 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-05-03 23:15:54,615 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(3094671d-7f0e-46fa-b383-23559a5cad4a, queue=dev-awx-0)
[2021-05-03 23:15:54,615: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(3094671d-7f0e-46fa-b383-23559a5cad4a, queue=dev-awx-0)
2021-05-03 23:15:54,653 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(caa09d43-cc70-4d15-9718-c28157e52110, queue=awx_private_queue)
[2021-05-03 23:15:54,653: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(caa09d43-cc70-4d15-9718-c28157e52110, queue=awx_private_queue)
2021-05-03 23:15:54,781 DEBUG awx.main.dispatch delivered 3094671d-7f0e-46fa-b383-23559a5cad4a to worker[202] qsize 0
2021-05-03 23:15:54,784 DEBUG awx.main.dispatch delivered caa09d43-cc70-4d15-9718-c28157e52110 to worker[195] qsize 0
2021-05-03 23:15:54,784 DEBUG awx.main.dispatch task 3094671d-7f0e-46fa-b383-23559a5cad4a starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:15:54,786 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:15:54,786 DEBUG awx.main.dispatch task caa09d43-cc70-4d15-9718-c28157e52110 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:15:54,788 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:15:54,792 DEBUG awx.main.dispatch task bfd905d5-e69b-47c7-9ae6-3ec4453a1ca1 is finished
2021-05-03 23:15:54,832 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:15:54,854 DEBUG awx.main.dispatch task 3094671d-7f0e-46fa-b383-23559a5cad4a is finished
2021-05-03 23:15:54,917 DEBUG awx.main.dispatch task caa09d43-cc70-4d15-9718-c28157e52110 is finished
RESULT 2
OKREADY
2021-05-03 23:16:05,449 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(9533ae2b-84b8-4def-906c-6971fbb6c7e1, queue=dev-awx-0)
[2021-05-03 23:16:05,449: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(9533ae2b-84b8-4def-906c-6971fbb6c7e1, queue=dev-awx-0)
2021-05-03 23:16:05,470 DEBUG awx.main.dispatch delivered 9533ae2b-84b8-4def-906c-6971fbb6c7e1 to worker[202] qsize 0
2021-05-03 23:16:05,474 DEBUG awx.main.dispatch task 9533ae2b-84b8-4def-906c-6971fbb6c7e1 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:16:05,519 DEBUG awx.main.dispatch task 9533ae2b-84b8-4def-906c-6971fbb6c7e1 is finished
2021-05-03 23:16:05,851 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(88806c6e-bb7e-408e-8d82-d524a26a88d2, queue=awx_private_queue)
[2021-05-03 23:16:05,851: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(88806c6e-bb7e-408e-8d82-d524a26a88d2, queue=awx_private_queue)
2021-05-03 23:16:05,868 DEBUG awx.main.dispatch delivered 88806c6e-bb7e-408e-8d82-d524a26a88d2 to worker[195] qsize 0
2021-05-03 23:16:05,872 DEBUG awx.main.dispatch task 88806c6e-bb7e-408e-8d82-d524a26a88d2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:16:05,900 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:16:05,923 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:15:42.054592+00:00
2021-05-03 23:16:05,981 DEBUG awx.main.dispatch task 88806c6e-bb7e-408e-8d82-d524a26a88d2 is finished
2021-05-03 23:16:14,654 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(357b4642-deda-4d5b-ad3e-0ea0b52438c1, queue=awx_private_queue)
[2021-05-03 23:16:14,654: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(357b4642-deda-4d5b-ad3e-0ea0b52438c1, queue=awx_private_queue)
2021-05-03 23:16:14,669 DEBUG awx.main.dispatch delivered 357b4642-deda-4d5b-ad3e-0ea0b52438c1 to worker[204] qsize 0
2021-05-03 23:16:14,672 DEBUG awx.main.dispatch task 357b4642-deda-4d5b-ad3e-0ea0b52438c1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:16:14,674 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:16:14,723 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:16:14,830 DEBUG awx.main.dispatch task 357b4642-deda-4d5b-ad3e-0ea0b52438c1 is finished
2021-05-03 23:16:34,667 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2bfd02a6-dd33-435a-86f1-e7831286637a, queue=awx_private_queue)
[2021-05-03 23:16:34,667: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2bfd02a6-dd33-435a-86f1-e7831286637a, queue=awx_private_queue)
2021-05-03 23:16:34,684 DEBUG awx.main.dispatch delivered 2bfd02a6-dd33-435a-86f1-e7831286637a to worker[195] qsize 0
2021-05-03 23:16:34,687 DEBUG awx.main.dispatch task 2bfd02a6-dd33-435a-86f1-e7831286637a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:16:34,688 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:16:35,851 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2a52f7b2-e605-4b21-ba6a-5df883598ae9, queue=awx_private_queue)
[2021-05-03 23:16:35,851: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2a52f7b2-e605-4b21-ba6a-5df883598ae9, queue=awx_private_queue)
2021-05-03 23:16:35,866 DEBUG awx.main.dispatch delivered 2a52f7b2-e605-4b21-ba6a-5df883598ae9 to worker[197] qsize 0
2021-05-03 23:16:35,870 DEBUG awx.main.dispatch task 2a52f7b2-e605-4b21-ba6a-5df883598ae9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:16:37,840 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:16:38,068 DEBUG awx.main.dispatch task 2bfd02a6-dd33-435a-86f1-e7831286637a is finished
2021-05-03 23:16:38,489 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:16:38,496 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:16:05.901205+00:00
2021-05-03 23:16:38,690 DEBUG awx.main.dispatch task 2a52f7b2-e605-4b21-ba6a-5df883598ae9 is finished
RESULT 2
OKREADY
2021-05-03 23:16:54,619 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c69f9d97-2fe2-4625-8b70-8406bb2d7953, queue=dev-awx-0)
[2021-05-03 23:16:54,619: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c69f9d97-2fe2-4625-8b70-8406bb2d7953, queue=dev-awx-0)
2021-05-03 23:17:32,664 DEBUG awx.main.dispatch delivered c69f9d97-2fe2-4625-8b70-8406bb2d7953 to worker[202] qsize 0
2021-05-03 23:17:32,858 DEBUG awx.main.dispatch task c69f9d97-2fe2-4625-8b70-8406bb2d7953 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:17:32,861 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:17:32,584 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c9ab8282-8b3d-4c31-9101-1df630fc9421, queue=awx_private_queue)
[2021-05-03 23:17:32,584: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c9ab8282-8b3d-4c31-9101-1df630fc9421, queue=awx_private_queue)
2021-05-03 23:17:32,982 DEBUG awx.main.dispatch delivered c9ab8282-8b3d-4c31-9101-1df630fc9421 to worker[204] qsize 0
2021-05-03 23:17:32,983 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2ab48664-afc1-4c45-88b8-2529cdec2dda, queue=dev-awx-0)
[2021-05-03 23:17:32,983: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2ab48664-afc1-4c45-88b8-2529cdec2dda, queue=dev-awx-0)
2021-05-03 23:17:32,986 DEBUG awx.main.dispatch task c9ab8282-8b3d-4c31-9101-1df630fc9421 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:17:32,991 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:17:32,999 DEBUG awx.main.dispatch delivered 2ab48664-afc1-4c45-88b8-2529cdec2dda to worker[195] qsize 0
2021-05-03 23:17:33,000 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ec2454ff-1578-4341-8580-6ec5fd8d1ad1, queue=awx_private_queue)
[2021-05-03 23:17:33,000: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ec2454ff-1578-4341-8580-6ec5fd8d1ad1, queue=awx_private_queue)
2021-05-03 23:17:33,008 DEBUG awx.main.dispatch task 2ab48664-afc1-4c45-88b8-2529cdec2dda starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:17:33,016 DEBUG awx.main.dispatch delivered ec2454ff-1578-4341-8580-6ec5fd8d1ad1 to worker[197] qsize 0
2021-05-03 23:17:33,020 DEBUG awx.main.dispatch task ec2454ff-1578-4341-8580-6ec5fd8d1ad1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:17:34,240 DEBUG awx.main.dispatch task 2ab48664-afc1-4c45-88b8-2529cdec2dda is finished
2021-05-03 23:17:34,813 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:17:34,822 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:16:38.489456+00:00
2021-05-03 23:17:34,933 DEBUG awx.main.dispatch task ec2454ff-1578-4341-8580-6ec5fd8d1ad1 is finished
2021-05-03 23:17:37,768 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:17:37,808 DEBUG awx.main.dispatch task c69f9d97-2fe2-4625-8b70-8406bb2d7953 is finished
2021-05-03 23:17:37,895 DEBUG awx.main.dispatch task c9ab8282-8b3d-4c31-9101-1df630fc9421 is finished
2021-05-03 23:17:52,584 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e1d03e07-a89a-4355-9d86-111907e74fc5, queue=awx_private_queue)
[2021-05-03 23:17:52,584: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e1d03e07-a89a-4355-9d86-111907e74fc5, queue=awx_private_queue)
2021-05-03 23:17:52,602 DEBUG awx.main.dispatch delivered e1d03e07-a89a-4355-9d86-111907e74fc5 to worker[204] qsize 0
2021-05-03 23:17:52,605 DEBUG awx.main.dispatch task e1d03e07-a89a-4355-9d86-111907e74fc5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:17:52,607 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:17:54,403 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:17:54,784 DEBUG awx.main.dispatch task e1d03e07-a89a-4355-9d86-111907e74fc5 is finished
RESULT 2
OKREADY
2021-05-03 23:18:03,001 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(666f7aa6-9f80-4bd5-93b6-58c24060c209, queue=awx_private_queue)
[2021-05-03 23:18:03,001: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(666f7aa6-9f80-4bd5-93b6-58c24060c209, queue=awx_private_queue)
2021-05-03 23:18:03,027 DEBUG awx.main.dispatch delivered 666f7aa6-9f80-4bd5-93b6-58c24060c209 to worker[195] qsize 0
2021-05-03 23:18:03,031 DEBUG awx.main.dispatch task 666f7aa6-9f80-4bd5-93b6-58c24060c209 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:18:12,584 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d739f773-56e4-415d-91ae-7699fb672fa6, queue=awx_private_queue)
[2021-05-03 23:18:12,584: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d739f773-56e4-415d-91ae-7699fb672fa6, queue=awx_private_queue)
2021-05-03 23:18:12,601 DEBUG awx.main.dispatch delivered d739f773-56e4-415d-91ae-7699fb672fa6 to worker[202] qsize 0
2021-05-03 23:18:12,602 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(83625c25-454c-4374-afe0-54cd15cd07a9, queue=dev-awx-0)
[2021-05-03 23:18:12,602: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(83625c25-454c-4374-afe0-54cd15cd07a9, queue=dev-awx-0)
2021-05-03 23:18:12,603 DEBUG awx.main.dispatch task d739f773-56e4-415d-91ae-7699fb672fa6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:18:12,605 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:18:12,977 DEBUG awx.main.dispatch delivered 83625c25-454c-4374-afe0-54cd15cd07a9 to worker[204] qsize 0
2021-05-03 23:18:12,980 DEBUG awx.main.dispatch task 83625c25-454c-4374-afe0-54cd15cd07a9 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:18:12,983 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:18:13,110 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:18:13,125 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:17:34.813741+00:00
2021-05-03 23:18:14,600 DEBUG awx.main.dispatch task 666f7aa6-9f80-4bd5-93b6-58c24060c209 is finished
2021-05-03 23:18:19,409 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:18:19,885 DEBUG awx.main.dispatch task d739f773-56e4-415d-91ae-7699fb672fa6 is finished
2021-05-03 23:18:21,453 DEBUG awx.main.dispatch task 83625c25-454c-4374-afe0-54cd15cd07a9 is finished
2021-05-03 23:18:32,590 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7ad2270b-3776-42f7-89c5-249ce3a1b54d, queue=awx_private_queue)
[2021-05-03 23:18:32,590: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7ad2270b-3776-42f7-89c5-249ce3a1b54d, queue=awx_private_queue)
2021-05-03 23:18:36,493 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(98ad89f4-4030-4fc9-b255-bd5fab15d684, queue=dev-awx-0)
[2021-05-03 23:18:36,493: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(98ad89f4-4030-4fc9-b255-bd5fab15d684, queue=dev-awx-0)
2021-05-03 23:18:36,493 DEBUG awx.main.dispatch delivered 7ad2270b-3776-42f7-89c5-249ce3a1b54d to worker[197] qsize 0
2021-05-03 23:18:36,501 DEBUG awx.main.dispatch task 7ad2270b-3776-42f7-89c5-249ce3a1b54d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:18:36,505 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:18:36,514 DEBUG awx.main.dispatch delivered 98ad89f4-4030-4fc9-b255-bd5fab15d684 to worker[202] qsize 0
2021-05-03 23:18:36,515 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(02965b66-799c-4011-9669-aa09400c0589, queue=awx_private_queue)
[2021-05-03 23:18:36,515: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(02965b66-799c-4011-9669-aa09400c0589, queue=awx_private_queue)
2021-05-03 23:18:36,517 DEBUG awx.main.dispatch task 98ad89f4-4030-4fc9-b255-bd5fab15d684 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:18:36,529 DEBUG awx.main.dispatch delivered 02965b66-799c-4011-9669-aa09400c0589 to worker[195] qsize 0
2021-05-03 23:18:36,532 DEBUG awx.main.dispatch task 02965b66-799c-4011-9669-aa09400c0589 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:18:36,586 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:18:36,593 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:18:13.111274+00:00
2021-05-03 23:18:36,600 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:18:36,614 DEBUG awx.main.dispatch task 98ad89f4-4030-4fc9-b255-bd5fab15d684 is finished
2021-05-03 23:18:36,641 DEBUG awx.main.dispatch task 02965b66-799c-4011-9669-aa09400c0589 is finished
2021-05-03 23:18:36,808 DEBUG awx.main.dispatch task 7ad2270b-3776-42f7-89c5-249ce3a1b54d is finished
2021-05-03 23:18:52,592 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f37e7dc7-828f-434a-a31e-552c8879db2d, queue=awx_private_queue)
[2021-05-03 23:18:52,592: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f37e7dc7-828f-434a-a31e-552c8879db2d, queue=awx_private_queue)
2021-05-03 23:18:52,611 DEBUG awx.main.dispatch delivered f37e7dc7-828f-434a-a31e-552c8879db2d to worker[204] qsize 0
2021-05-03 23:18:52,614 DEBUG awx.main.dispatch task f37e7dc7-828f-434a-a31e-552c8879db2d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:18:52,616 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:18:53,625 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:18:53,681 DEBUG awx.main.dispatch task f37e7dc7-828f-434a-a31e-552c8879db2d is finished
RESULT 2
OKREADY
2021-05-03 23:19:06,515 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(39999bf0-1e90-4740-972f-067ec0bbfc83, queue=awx_private_queue)
[2021-05-03 23:19:06,515: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(39999bf0-1e90-4740-972f-067ec0bbfc83, queue=awx_private_queue)
2021-05-03 23:19:06,547 DEBUG awx.main.dispatch delivered 39999bf0-1e90-4740-972f-067ec0bbfc83 to worker[202] qsize 0
2021-05-03 23:19:06,549 DEBUG awx.main.dispatch task 39999bf0-1e90-4740-972f-067ec0bbfc83 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:19:06,643 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:19:06,657 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:18:36.586605+00:00
2021-05-03 23:19:06,687 DEBUG awx.main.dispatch task 39999bf0-1e90-4740-972f-067ec0bbfc83 is finished
2021-05-03 23:19:12,591 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9ca114a9-00bb-4f3c-97a8-68b5216bcda0, queue=awx_private_queue)
[2021-05-03 23:19:12,591: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9ca114a9-00bb-4f3c-97a8-68b5216bcda0, queue=awx_private_queue)
2021-05-03 23:19:12,607 DEBUG awx.main.dispatch delivered 9ca114a9-00bb-4f3c-97a8-68b5216bcda0 to worker[195] qsize 0
2021-05-03 23:19:12,608 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5146f0ec-24bd-49db-94da-4836e6980262, queue=dev-awx-0)
[2021-05-03 23:19:12,608: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5146f0ec-24bd-49db-94da-4836e6980262, queue=dev-awx-0)
2021-05-03 23:19:12,609 DEBUG awx.main.dispatch task 9ca114a9-00bb-4f3c-97a8-68b5216bcda0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:19:12,611 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:19:12,659 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:19:12,688 DEBUG awx.main.dispatch delivered 5146f0ec-24bd-49db-94da-4836e6980262 to worker[204] qsize 0
2021-05-03 23:19:12,690 DEBUG awx.main.dispatch task 5146f0ec-24bd-49db-94da-4836e6980262 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:19:12,691 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:19:12,703 DEBUG awx.main.dispatch task 9ca114a9-00bb-4f3c-97a8-68b5216bcda0 is finished
2021-05-03 23:19:12,755 DEBUG awx.main.dispatch task 5146f0ec-24bd-49db-94da-4836e6980262 is finished
2021-05-03 23:19:32,591 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fd11f988-58c9-4ada-b3f9-683fe9c7b2d1, queue=awx_private_queue)
[2021-05-03 23:19:32,591: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fd11f988-58c9-4ada-b3f9-683fe9c7b2d1, queue=awx_private_queue)
2021-05-03 23:19:32,608 DEBUG awx.main.dispatch delivered fd11f988-58c9-4ada-b3f9-683fe9c7b2d1 to worker[195] qsize 0
2021-05-03 23:19:32,612 DEBUG awx.main.dispatch task fd11f988-58c9-4ada-b3f9-683fe9c7b2d1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:19:32,617 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:19:32,670 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:19:32,724 DEBUG awx.main.dispatch task fd11f988-58c9-4ada-b3f9-683fe9c7b2d1 is finished
2021-05-03 23:19:36,493 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(65915e5f-65fb-469b-9ed0-770e5eeea790, queue=dev-awx-0)
[2021-05-03 23:19:36,493: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(65915e5f-65fb-469b-9ed0-770e5eeea790, queue=dev-awx-0)
2021-05-03 23:19:37,255 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bd2c13b7-4020-4432-9d0d-58a50f472142, queue=awx_private_queue)
[2021-05-03 23:19:37,255: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bd2c13b7-4020-4432-9d0d-58a50f472142, queue=awx_private_queue)
2021-05-03 23:19:37,253 DEBUG awx.main.dispatch delivered 65915e5f-65fb-469b-9ed0-770e5eeea790 to worker[204] qsize 0
2021-05-03 23:19:37,434 DEBUG awx.main.dispatch delivered bd2c13b7-4020-4432-9d0d-58a50f472142 to worker[202] qsize 0
2021-05-03 23:19:37,434 DEBUG awx.main.dispatch task 65915e5f-65fb-469b-9ed0-770e5eeea790 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:19:37,437 DEBUG awx.main.dispatch task bd2c13b7-4020-4432-9d0d-58a50f472142 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:19:37,472 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:19:37,477 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:19:06.643886+00:00
2021-05-03 23:19:37,528 DEBUG awx.main.dispatch task bd2c13b7-4020-4432-9d0d-58a50f472142 is finished
2021-05-03 23:19:37,565 DEBUG awx.main.dispatch task 65915e5f-65fb-469b-9ed0-770e5eeea790 is finished
2021-05-03 23:19:52,597 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d7f3e6d8-57eb-4ca2-b810-51d81408461e, queue=awx_private_queue)
[2021-05-03 23:19:52,597: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d7f3e6d8-57eb-4ca2-b810-51d81408461e, queue=awx_private_queue)
2021-05-03 23:19:52,613 DEBUG awx.main.dispatch delivered d7f3e6d8-57eb-4ca2-b810-51d81408461e to worker[197] qsize 0
2021-05-03 23:19:52,618 DEBUG awx.main.dispatch task d7f3e6d8-57eb-4ca2-b810-51d81408461e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:19:52,624 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:19:54,477 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:19:54,522 DEBUG awx.main.dispatch task d7f3e6d8-57eb-4ca2-b810-51d81408461e is finished
RESULT 2
OKREADY
2021-05-03 23:20:07,265 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(becae3d0-ac3a-492d-a7a9-7be5318f6b09, queue=awx_private_queue)
[2021-05-03 23:20:07,265: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(becae3d0-ac3a-492d-a7a9-7be5318f6b09, queue=awx_private_queue)
2021-05-03 23:20:07,291 DEBUG awx.main.dispatch delivered becae3d0-ac3a-492d-a7a9-7be5318f6b09 to worker[204] qsize 0
2021-05-03 23:20:07,294 DEBUG awx.main.dispatch task becae3d0-ac3a-492d-a7a9-7be5318f6b09 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:20:12,600 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0979f231-a678-427b-9702-15187c582f98, queue=awx_private_queue)
[2021-05-03 23:20:12,600: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0979f231-a678-427b-9702-15187c582f98, queue=awx_private_queue)
2021-05-03 23:20:12,620 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(ec8d9584-b835-4e88-a068-1ffbde1b0033, queue=dev-awx-0)
[2021-05-03 23:20:12,620: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(ec8d9584-b835-4e88-a068-1ffbde1b0033, queue=dev-awx-0)
2021-05-03 23:20:12,619 DEBUG awx.main.dispatch delivered 0979f231-a678-427b-9702-15187c582f98 to worker[195] qsize 0
2021-05-03 23:20:12,624 DEBUG awx.main.dispatch task 0979f231-a678-427b-9702-15187c582f98 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:20:12,627 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:20:18,080 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:20:18,097 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:19:37.473011+00:00
2021-05-03 23:20:18,240 DEBUG awx.main.dispatch task becae3d0-ac3a-492d-a7a9-7be5318f6b09 is finished
2021-05-03 23:20:19,997 DEBUG awx.main.dispatch delivered ec8d9584-b835-4e88-a068-1ffbde1b0033 to worker[204] qsize 0
2021-05-03 23:20:20,001 DEBUG awx.main.dispatch task ec8d9584-b835-4e88-a068-1ffbde1b0033 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:20:20,003 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:20:30,135 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:20:30,220 DEBUG awx.main.dispatch task ec8d9584-b835-4e88-a068-1ffbde1b0033 is finished
2021-05-03 23:20:32,604 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(22405b21-cd90-40be-a0e5-3825ddb53147, queue=awx_private_queue)
[2021-05-03 23:20:32,604: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(22405b21-cd90-40be-a0e5-3825ddb53147, queue=awx_private_queue)
2021-05-03 23:20:32,634 DEBUG awx.main.dispatch delivered 22405b21-cd90-40be-a0e5-3825ddb53147 to worker[204] qsize 0
2021-05-03 23:20:32,636 DEBUG awx.main.dispatch task 22405b21-cd90-40be-a0e5-3825ddb53147 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:20:32,638 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:20:34,780 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-03 23:20:34,780 DEBUG awx.main.dispatch task 22405b21-cd90-40be-a0e5-3825ddb53147 is finished
2021-05-03 23:20:36,494 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(fda1f458-16ad-4e52-b94b-32be5d94393d, queue=dev-awx-0)
[2021-05-03 23:20:36,494: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(fda1f458-16ad-4e52-b94b-32be5d94393d, queue=dev-awx-0)
2021-05-03 23:20:36,517 DEBUG awx.main.dispatch delivered fda1f458-16ad-4e52-b94b-32be5d94393d to worker[197] qsize 0
2021-05-03 23:20:36,520 DEBUG awx.main.dispatch task fda1f458-16ad-4e52-b94b-32be5d94393d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:20:37,265 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(068b1a1f-aca5-469f-8343-e0f11f3e425b, queue=awx_private_queue)
[2021-05-03 23:20:37,265: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(068b1a1f-aca5-469f-8343-e0f11f3e425b, queue=awx_private_queue)
2021-05-03 23:20:37,289 DEBUG awx.main.dispatch delivered 068b1a1f-aca5-469f-8343-e0f11f3e425b to worker[202] qsize 0
2021-05-03 23:20:37,725 DEBUG awx.main.dispatch task fda1f458-16ad-4e52-b94b-32be5d94393d is finished
2021-05-03 23:20:37,294 DEBUG awx.main.dispatch task 068b1a1f-aca5-469f-8343-e0f11f3e425b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:20:38,636 DEBUG awx.main.dispatch task 0979f231-a678-427b-9702-15187c582f98 is finished
2021-05-03 23:20:39,449 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:20:39,485 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:20:18.081252+00:00
2021-05-03 23:20:49,854 DEBUG awx.main.dispatch task 068b1a1f-aca5-469f-8343-e0f11f3e425b is finished
2021-05-03 23:20:52,604 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9d0dfc84-53a0-4002-8e0f-7c7b2c258b60, queue=awx_private_queue)
[2021-05-03 23:20:52,604: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9d0dfc84-53a0-4002-8e0f-7c7b2c258b60, queue=awx_private_queue)
2021-05-03 23:20:52,624 DEBUG awx.main.dispatch delivered 9d0dfc84-53a0-4002-8e0f-7c7b2c258b60 to worker[204] qsize 0
2021-05-03 23:20:52,626 DEBUG awx.main.dispatch task 9d0dfc84-53a0-4002-8e0f-7c7b2c258b60 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:20:52,627 DEBUG awx.main.scheduler Running Tower task manager.
RESULT 2
OKREADY
2021-05-03 23:21:07,270 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a5cb090e-9bcb-4d65-96df-39831a45eb1c, queue=awx_private_queue)
[2021-05-03 23:21:07,270: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a5cb090e-9bcb-4d65-96df-39831a45eb1c, queue=awx_private_queue)
2021-05-03 23:21:07,286 DEBUG awx.main.dispatch delivered a5cb090e-9bcb-4d65-96df-39831a45eb1c to worker[197] qsize 0
2021-05-03 23:21:07,288 DEBUG awx.main.dispatch task a5cb090e-9bcb-4d65-96df-39831a45eb1c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:21:12,604 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b5f27fc9-596d-4984-bc56-9a23e88fcb35, queue=awx_private_queue)
[2021-05-03 23:21:12,604: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b5f27fc9-596d-4984-bc56-9a23e88fcb35, queue=awx_private_queue)
2021-05-03 23:21:12,625 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a2bb5136-1504-40f2-bf10-e8c37716c137, queue=dev-awx-0)
[2021-05-03 23:21:12,625: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a2bb5136-1504-40f2-bf10-e8c37716c137, queue=dev-awx-0)
2021-05-03 23:21:12,624 DEBUG awx.main.dispatch delivered b5f27fc9-596d-4984-bc56-9a23e88fcb35 to worker[202] qsize 0
2021-05-03 23:21:12,628 DEBUG awx.main.dispatch task b5f27fc9-596d-4984-bc56-9a23e88fcb35 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:21:12,631 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:21:20,676 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:21:22,066 DEBUG awx.main.dispatch task 9d0dfc84-53a0-4002-8e0f-7c7b2c258b60 is finished
2021-05-03 23:21:23,342 DEBUG awx.main.dispatch delivered a2bb5136-1504-40f2-bf10-e8c37716c137 to worker[204] qsize 0
2021-05-03 23:21:23,345 DEBUG awx.main.dispatch task a2bb5136-1504-40f2-bf10-e8c37716c137 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:21:23,346 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:21:24,074 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:21:24,089 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:20:39.450245+00:00
2021-05-03 23:21:27,397 DEBUG awx.main.dispatch task a5cb090e-9bcb-4d65-96df-39831a45eb1c is finished
2021-05-03 23:21:32,613 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(04d2a93b-e5f9-40b3-a91d-f9387c12d6f6, queue=awx_private_queue)
[2021-05-03 23:21:32,613: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(04d2a93b-e5f9-40b3-a91d-f9387c12d6f6, queue=awx_private_queue)
2021-05-03 23:21:32,638 DEBUG awx.main.dispatch delivered 04d2a93b-e5f9-40b3-a91d-f9387c12d6f6 to worker[197] qsize 0
2021-05-03 23:21:32,641 DEBUG awx.main.dispatch task 04d2a93b-e5f9-40b3-a91d-f9387c12d6f6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:21:32,643 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:21:36,494 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(595ee45b-ce87-4fcf-a935-a36d37ebb80e, queue=dev-awx-0)
[2021-05-03 23:21:36,494: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(595ee45b-ce87-4fcf-a935-a36d37ebb80e, queue=dev-awx-0)
2021-05-03 23:21:36,521 DEBUG awx.main.dispatch delivered 595ee45b-ce87-4fcf-a935-a36d37ebb80e to worker[195] qsize 0
2021-05-03 23:21:36,527 DEBUG awx.main.dispatch task 595ee45b-ce87-4fcf-a935-a36d37ebb80e starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:21:37,270 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(39eb9bab-8680-425a-bfef-a9b0865b76fe, queue=awx_private_queue)
[2021-05-03 23:21:37,270: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(39eb9bab-8680-425a-bfef-a9b0865b76fe, queue=awx_private_queue)
2021-05-03 23:21:37,321 WARNING awx.main.dispatch scaling up worker pid:208
2021-05-03 23:21:37,329 DEBUG awx.main.dispatch delivered 39eb9bab-8680-425a-bfef-a9b0865b76fe to worker[208] qsize 0
2021-05-03 23:21:37,334 DEBUG awx.main.dispatch task 39eb9bab-8680-425a-bfef-a9b0865b76fe starting awx.main.tasks.awx_periodic_scheduler(*[])
RESULT 2
OKREADY
2021-05-03 23:22:07,978 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:22:10,506 DEBUG awx.main.dispatch task 595ee45b-ce87-4fcf-a935-a36d37ebb80e is finished
2021-05-03 23:22:17,009 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-03 23:22:17,009 DEBUG awx.main.dispatch task 04d2a93b-e5f9-40b3-a91d-f9387c12d6f6 is finished
2021-05-03 23:22:26,292 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:22:26,293 DEBUG awx.main.dispatch task a2bb5136-1504-40f2-bf10-e8c37716c137 is finished
2021-05-03 23:22:27,430 DEBUG awx.main.dispatch task b5f27fc9-596d-4984-bc56-9a23e88fcb35 is finished
2021-05-03 23:22:28,433 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:21:24.075667+00:00
2021-05-03 23:22:46,677 DEBUG awx.main.dispatch task 39eb9bab-8680-425a-bfef-a9b0865b76fe is finished
RESULT 2
OKREADY
2021-05-03 23:21:52,623 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(851892b4-eb19-4821-8698-ab2d78d6b918, queue=awx_private_queue)
[2021-05-03 23:21:52,623: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(851892b4-eb19-4821-8698-ab2d78d6b918, queue=awx_private_queue)
2021-05-03 23:23:07,331 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(936f4f41-6604-4580-a280-774b8069f03a, queue=awx_private_queue)
[2021-05-03 23:23:07,331: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(936f4f41-6604-4580-a280-774b8069f03a, queue=awx_private_queue)
2021-05-03 23:23:07,333 DEBUG awx.main.dispatch delivered 851892b4-eb19-4821-8698-ab2d78d6b918 to worker[204] qsize 0
2021-05-03 23:23:07,341 DEBUG awx.main.dispatch task 851892b4-eb19-4821-8698-ab2d78d6b918 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:23:07,343 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:23:07,353 DEBUG awx.main.dispatch delivered 936f4f41-6604-4580-a280-774b8069f03a to worker[197] qsize 0
2021-05-03 23:23:07,356 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(20777fca-8c54-4e41-a9ab-73b322ab55a5, queue=dev-awx-0)
[2021-05-03 23:23:07,356: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(20777fca-8c54-4e41-a9ab-73b322ab55a5, queue=dev-awx-0)
2021-05-03 23:23:07,358 DEBUG awx.main.dispatch task 936f4f41-6604-4580-a280-774b8069f03a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:23:07,400 WARNING awx.main.dispatch scaling down worker pid:195
2021-05-03 23:23:07,401 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(1cb4ea47-08c3-4c69-bb07-81de58bb617c, queue=dev-awx-0)
[2021-05-03 23:23:07,401: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(1cb4ea47-08c3-4c69-bb07-81de58bb617c, queue=dev-awx-0)
2021-05-03 23:23:07,407 WARNING awx.main.dispatch worker exiting gracefully pid:195
2021-05-03 23:23:07,419 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5d93f426-4bbd-4b05-a0d7-d97e0f2d4bdd, queue=awx_private_queue)
[2021-05-03 23:23:07,419: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5d93f426-4bbd-4b05-a0d7-d97e0f2d4bdd, queue=awx_private_queue)
2021-05-03 23:23:11,099 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:23:11,181 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:22:26.294083+00:00
2021-05-03 23:23:11,716 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:23:11,785 DEBUG awx.main.dispatch task 851892b4-eb19-4821-8698-ab2d78d6b918 is finished
2021-05-03 23:23:13,823 DEBUG awx.main.dispatch delivered 20777fca-8c54-4e41-a9ab-73b322ab55a5 to worker[204] qsize 0
2021-05-03 23:23:13,826 DEBUG awx.main.dispatch delivered 1cb4ea47-08c3-4c69-bb07-81de58bb617c to worker[208] qsize 0
2021-05-03 23:23:13,826 DEBUG awx.main.dispatch task 20777fca-8c54-4e41-a9ab-73b322ab55a5 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:23:13,828 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:23:13,828 DEBUG awx.main.dispatch delivered 5d93f426-4bbd-4b05-a0d7-d97e0f2d4bdd to worker[202] qsize 0
2021-05-03 23:23:13,828 DEBUG awx.main.dispatch task 1cb4ea47-08c3-4c69-bb07-81de58bb617c starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:23:13,831 DEBUG awx.main.dispatch task 5d93f426-4bbd-4b05-a0d7-d97e0f2d4bdd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:23:13,835 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:23:15,044 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:23:15,050 DEBUG awx.main.dispatch task 1cb4ea47-08c3-4c69-bb07-81de58bb617c is finished
2021-05-03 23:23:15,101 DEBUG awx.main.dispatch task 5d93f426-4bbd-4b05-a0d7-d97e0f2d4bdd is finished
2021-05-03 23:23:16,609 DEBUG awx.main.dispatch task 20777fca-8c54-4e41-a9ab-73b322ab55a5 is finished
2021-05-03 23:23:17,538 DEBUG awx.main.dispatch task 936f4f41-6604-4580-a280-774b8069f03a is finished
2021-05-03 23:23:27,434 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(60713b09-98d9-48ee-ae7d-1052f3db7e2d, queue=awx_private_queue)
[2021-05-03 23:23:27,434: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(60713b09-98d9-48ee-ae7d-1052f3db7e2d, queue=awx_private_queue)
2021-05-03 23:23:27,460 DEBUG awx.main.dispatch delivered 60713b09-98d9-48ee-ae7d-1052f3db7e2d to worker[204] qsize 0
2021-05-03 23:23:27,464 DEBUG awx.main.dispatch task 60713b09-98d9-48ee-ae7d-1052f3db7e2d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:23:27,744 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:23:27,789 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:23:27,880 DEBUG awx.main.dispatch task 60713b09-98d9-48ee-ae7d-1052f3db7e2d is finished
2021-05-03 23:23:37,336 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(fd2e76c1-4e35-4d4b-8421-2305d2de4e4d, queue=awx_private_queue)
[2021-05-03 23:23:37,336: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(fd2e76c1-4e35-4d4b-8421-2305d2de4e4d, queue=awx_private_queue)
2021-05-03 23:23:37,364 DEBUG awx.main.dispatch delivered fd2e76c1-4e35-4d4b-8421-2305d2de4e4d to worker[197] qsize 0
2021-05-03 23:23:37,369 DEBUG awx.main.dispatch task fd2e76c1-4e35-4d4b-8421-2305d2de4e4d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:23:38,083 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:23:38,090 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:23:11.099665+00:00
2021-05-03 23:23:38,731 DEBUG awx.main.dispatch task fd2e76c1-4e35-4d4b-8421-2305d2de4e4d is finished
2021-05-03 23:23:47,433 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d4a69b72-5b05-4d42-9b43-b53cb289170c, queue=awx_private_queue)
[2021-05-03 23:23:47,433: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d4a69b72-5b05-4d42-9b43-b53cb289170c, queue=awx_private_queue)
2021-05-03 23:23:47,451 DEBUG awx.main.dispatch delivered d4a69b72-5b05-4d42-9b43-b53cb289170c to worker[208] qsize 0
2021-05-03 23:23:47,457 DEBUG awx.main.dispatch task d4a69b72-5b05-4d42-9b43-b53cb289170c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:23:47,460 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:23:48,321 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:23:51,590 DEBUG awx.main.dispatch task d4a69b72-5b05-4d42-9b43-b53cb289170c is finished
RESULT 2
OKREADY
2021-05-03 23:24:07,342 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(31202aa6-6906-462d-9732-fda675a335e9, queue=awx_private_queue)
[2021-05-03 23:24:07,342: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(31202aa6-6906-462d-9732-fda675a335e9, queue=awx_private_queue)
2021-05-03 23:24:07,369 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(713db42a-3ecf-4418-87fa-6503d3d588ce, queue=dev-awx-0)
[2021-05-03 23:24:07,369: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(713db42a-3ecf-4418-87fa-6503d3d588ce, queue=dev-awx-0)
2021-05-03 23:24:07,368 DEBUG awx.main.dispatch delivered 31202aa6-6906-462d-9732-fda675a335e9 to worker[197] qsize 0
2021-05-03 23:24:07,373 DEBUG awx.main.dispatch task 31202aa6-6906-462d-9732-fda675a335e9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:24:07,401 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a4e7eb33-56ab-4df2-b91f-614638eec79f, queue=dev-awx-0)
[2021-05-03 23:24:07,401: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a4e7eb33-56ab-4df2-b91f-614638eec79f, queue=dev-awx-0)
2021-05-03 23:24:07,432 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f52051dc-5dff-46a5-8cdb-05333c10a21b, queue=awx_private_queue)
[2021-05-03 23:24:07,432: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f52051dc-5dff-46a5-8cdb-05333c10a21b, queue=awx_private_queue)
2021-05-03 23:24:12,080 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:24:12,109 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:23:38.083363+00:00
2021-05-03 23:24:12,169 DEBUG awx.main.dispatch delivered 713db42a-3ecf-4418-87fa-6503d3d588ce to worker[204] qsize 0
2021-05-03 23:24:12,171 DEBUG awx.main.dispatch delivered a4e7eb33-56ab-4df2-b91f-614638eec79f to worker[208] qsize 0
2021-05-03 23:24:12,173 DEBUG awx.main.dispatch delivered f52051dc-5dff-46a5-8cdb-05333c10a21b to worker[202] qsize 0
2021-05-03 23:24:12,173 DEBUG awx.main.dispatch task a4e7eb33-56ab-4df2-b91f-614638eec79f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:24:12,173 DEBUG awx.main.dispatch task 713db42a-3ecf-4418-87fa-6503d3d588ce starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:24:12,177 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:24:12,176 DEBUG awx.main.dispatch task f52051dc-5dff-46a5-8cdb-05333c10a21b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:24:12,180 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:24:12,370 DEBUG awx.main.dispatch task a4e7eb33-56ab-4df2-b91f-614638eec79f is finished
2021-05-03 23:24:12,391 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:24:12,426 DEBUG awx.main.dispatch task 713db42a-3ecf-4418-87fa-6503d3d588ce is finished
2021-05-03 23:24:12,449 DEBUG awx.main.dispatch task f52051dc-5dff-46a5-8cdb-05333c10a21b is finished
2021-05-03 23:24:12,461 DEBUG awx.main.dispatch task 31202aa6-6906-462d-9732-fda675a335e9 is finished
2021-05-03 23:24:27,434 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f583ae15-0a4c-4624-b106-59df1105a229, queue=awx_private_queue)
[2021-05-03 23:24:27,434: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f583ae15-0a4c-4624-b106-59df1105a229, queue=awx_private_queue)
2021-05-03 23:24:27,452 DEBUG awx.main.dispatch delivered f583ae15-0a4c-4624-b106-59df1105a229 to worker[197] qsize 0
2021-05-03 23:24:27,457 DEBUG awx.main.dispatch task f583ae15-0a4c-4624-b106-59df1105a229 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:24:27,462 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:24:27,583 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:24:27,630 DEBUG awx.main.dispatch task f583ae15-0a4c-4624-b106-59df1105a229 is finished
2021-05-03 23:24:37,342 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(217bf675-6775-4778-a100-c316196dd53d, queue=awx_private_queue)
[2021-05-03 23:24:37,342: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(217bf675-6775-4778-a100-c316196dd53d, queue=awx_private_queue)
2021-05-03 23:24:37,387 DEBUG awx.main.dispatch delivered 217bf675-6775-4778-a100-c316196dd53d to worker[202] qsize 0
2021-05-03 23:24:37,390 DEBUG awx.main.dispatch task 217bf675-6775-4778-a100-c316196dd53d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:24:37,400 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:24:37,403 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:24:12.080825+00:00
2021-05-03 23:24:37,452 DEBUG awx.main.dispatch task 217bf675-6775-4778-a100-c316196dd53d is finished
2021-05-03 23:24:47,434 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9cc5ce58-4728-4196-94a4-154c383c92ac, queue=awx_private_queue)
[2021-05-03 23:24:47,434: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9cc5ce58-4728-4196-94a4-154c383c92ac, queue=awx_private_queue)
2021-05-03 23:24:47,452 DEBUG awx.main.dispatch delivered 9cc5ce58-4728-4196-94a4-154c383c92ac to worker[208] qsize 0
2021-05-03 23:24:47,456 DEBUG awx.main.dispatch task 9cc5ce58-4728-4196-94a4-154c383c92ac starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:24:47,459 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:24:47,496 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:24:47,540 DEBUG awx.main.dispatch task 9cc5ce58-4728-4196-94a4-154c383c92ac is finished
RESULT 2
OKREADY
2021-05-03 23:25:07,341 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(28aefc59-29a9-4951-b8ed-d44ac5a13875, queue=awx_private_queue)
[2021-05-03 23:25:07,341: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(28aefc59-29a9-4951-b8ed-d44ac5a13875, queue=awx_private_queue)
2021-05-03 23:25:07,356 DEBUG awx.main.dispatch delivered 28aefc59-29a9-4951-b8ed-d44ac5a13875 to worker[204] qsize 0
2021-05-03 23:25:07,358 DEBUG awx.main.dispatch task 28aefc59-29a9-4951-b8ed-d44ac5a13875 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:25:07,369 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(77fdb6c1-b19f-4d2b-88e7-7188d2eb5486, queue=dev-awx-0)
[2021-05-03 23:25:07,369: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(77fdb6c1-b19f-4d2b-88e7-7188d2eb5486, queue=dev-awx-0)
2021-05-03 23:25:07,375 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:25:07,379 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:24:37.400527+00:00
2021-05-03 23:25:07,398 DEBUG awx.main.dispatch task 28aefc59-29a9-4951-b8ed-d44ac5a13875 is finished
2021-05-03 23:25:07,401 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(4ab2b192-b874-44da-ad7d-93b738d1d34f, queue=dev-awx-0)
[2021-05-03 23:25:07,401: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(4ab2b192-b874-44da-ad7d-93b738d1d34f, queue=dev-awx-0)
2021-05-03 23:25:07,403 DEBUG awx.main.dispatch delivered 77fdb6c1-b19f-4d2b-88e7-7188d2eb5486 to worker[208] qsize 0
2021-05-03 23:25:07,406 DEBUG awx.main.dispatch task 77fdb6c1-b19f-4d2b-88e7-7188d2eb5486 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:25:07,407 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:25:07,434 DEBUG awx.main.dispatch task 77fdb6c1-b19f-4d2b-88e7-7188d2eb5486 is finished
2021-05-03 23:25:07,434 DEBUG awx.main.dispatch delivered 4ab2b192-b874-44da-ad7d-93b738d1d34f to worker[204] qsize 0
2021-05-03 23:25:07,436 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0591b01a-6027-469e-a2ec-c431287e408c, queue=awx_private_queue)
[2021-05-03 23:25:07,436: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0591b01a-6027-469e-a2ec-c431287e408c, queue=awx_private_queue)
2021-05-03 23:25:07,436 DEBUG awx.main.dispatch task 4ab2b192-b874-44da-ad7d-93b738d1d34f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:25:07,450 DEBUG awx.main.dispatch delivered 0591b01a-6027-469e-a2ec-c431287e408c to worker[208] qsize 0
2021-05-03 23:25:07,452 DEBUG awx.main.dispatch task 0591b01a-6027-469e-a2ec-c431287e408c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:25:07,453 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:25:07,468 DEBUG awx.main.dispatch task 4ab2b192-b874-44da-ad7d-93b738d1d34f is finished
2021-05-03 23:25:07,476 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:25:07,505 DEBUG awx.main.dispatch task 0591b01a-6027-469e-a2ec-c431287e408c is finished
2021-05-03 23:25:27,445 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bb0a5225-08d7-42c3-8976-f118330723a7, queue=awx_private_queue)
[2021-05-03 23:25:27,445: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bb0a5225-08d7-42c3-8976-f118330723a7, queue=awx_private_queue)
2021-05-03 23:25:27,474 DEBUG awx.main.dispatch delivered bb0a5225-08d7-42c3-8976-f118330723a7 to worker[208] qsize 0
2021-05-03 23:25:27,479 DEBUG awx.main.dispatch task bb0a5225-08d7-42c3-8976-f118330723a7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:25:27,482 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:25:27,505 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:25:27,547 DEBUG awx.main.dispatch task bb0a5225-08d7-42c3-8976-f118330723a7 is finished
2021-05-03 23:25:37,342 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(db677e7c-b588-47d7-822c-5d4a178f8083, queue=awx_private_queue)
[2021-05-03 23:25:37,342: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(db677e7c-b588-47d7-822c-5d4a178f8083, queue=awx_private_queue)
2021-05-03 23:25:37,372 DEBUG awx.main.dispatch delivered db677e7c-b588-47d7-822c-5d4a178f8083 to worker[204] qsize 0
2021-05-03 23:25:37,379 DEBUG awx.main.dispatch task db677e7c-b588-47d7-822c-5d4a178f8083 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:25:37,394 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:25:37,400 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:25:07.376013+00:00
2021-05-03 23:25:37,415 DEBUG awx.main.dispatch task db677e7c-b588-47d7-822c-5d4a178f8083 is finished
2021-05-03 23:25:47,439 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(90f20281-a443-47d4-be39-da7be337add5, queue=awx_private_queue)
[2021-05-03 23:25:47,439: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(90f20281-a443-47d4-be39-da7be337add5, queue=awx_private_queue)
2021-05-03 23:25:47,471 DEBUG awx.main.dispatch delivered 90f20281-a443-47d4-be39-da7be337add5 to worker[197] qsize 0
2021-05-03 23:25:47,475 DEBUG awx.main.dispatch task 90f20281-a443-47d4-be39-da7be337add5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:25:47,479 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:25:47,502 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:25:47,538 DEBUG awx.main.dispatch task 90f20281-a443-47d4-be39-da7be337add5 is finished
2021-05-03 23:25:54,484 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(405ab105-cf08-4067-bd50-1421afe10d2e, queue=dev-awx-0)
[2021-05-03 23:25:54,484: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(405ab105-cf08-4067-bd50-1421afe10d2e, queue=dev-awx-0)
2021-05-03 23:25:54,504 DEBUG awx.main.dispatch delivered 405ab105-cf08-4067-bd50-1421afe10d2e to worker[202] qsize 0
2021-05-03 23:25:54,510 DEBUG awx.main.dispatch task 405ab105-cf08-4067-bd50-1421afe10d2e starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-05-03 23:25:54,731 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-05-03 23:25:54,763 DEBUG awx.main.dispatch task 405ab105-cf08-4067-bd50-1421afe10d2e is finished
RESULT 2
OKREADY
2021-05-03 23:26:07,344 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2e716bcf-2e98-49ed-9971-52b42a679775, queue=awx_private_queue)
[2021-05-03 23:26:07,344: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2e716bcf-2e98-49ed-9971-52b42a679775, queue=awx_private_queue)
2021-05-03 23:26:07,359 DEBUG awx.main.dispatch delivered 2e716bcf-2e98-49ed-9971-52b42a679775 to worker[204] qsize 0
2021-05-03 23:26:07,362 DEBUG awx.main.dispatch task 2e716bcf-2e98-49ed-9971-52b42a679775 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:26:07,369 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c51c9a08-7fd7-486a-9030-9c4c1f12e236, queue=dev-awx-0)
[2021-05-03 23:26:07,369: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c51c9a08-7fd7-486a-9030-9c4c1f12e236, queue=dev-awx-0)
2021-05-03 23:26:07,372 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:26:07,375 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:25:37.394934+00:00
2021-05-03 23:26:07,388 DEBUG awx.main.dispatch task 2e716bcf-2e98-49ed-9971-52b42a679775 is finished
2021-05-03 23:26:07,401 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(131b5482-96c4-41f6-aca5-f13b77e881a6, queue=dev-awx-0)
[2021-05-03 23:26:07,401: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(131b5482-96c4-41f6-aca5-f13b77e881a6, queue=dev-awx-0)
2021-05-03 23:26:07,402 DEBUG awx.main.dispatch delivered c51c9a08-7fd7-486a-9030-9c4c1f12e236 to worker[202] qsize 0
2021-05-03 23:26:07,404 DEBUG awx.main.dispatch task c51c9a08-7fd7-486a-9030-9c4c1f12e236 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:26:07,406 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:26:07,418 DEBUG awx.main.dispatch delivered 131b5482-96c4-41f6-aca5-f13b77e881a6 to worker[197] qsize 0
2021-05-03 23:26:07,421 DEBUG awx.main.dispatch task 131b5482-96c4-41f6-aca5-f13b77e881a6 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:26:07,433 DEBUG awx.main.dispatch task 131b5482-96c4-41f6-aca5-f13b77e881a6 is finished
2021-05-03 23:26:07,437 DEBUG awx.main.dispatch task c51c9a08-7fd7-486a-9030-9c4c1f12e236 is finished
2021-05-03 23:26:07,439 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(53b785d7-740a-477d-bc1d-8b2906a25bf3, queue=awx_private_queue)
[2021-05-03 23:26:07,439: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(53b785d7-740a-477d-bc1d-8b2906a25bf3, queue=awx_private_queue)
2021-05-03 23:26:07,452 DEBUG awx.main.dispatch delivered 53b785d7-740a-477d-bc1d-8b2906a25bf3 to worker[202] qsize 0
2021-05-03 23:26:07,454 DEBUG awx.main.dispatch task 53b785d7-740a-477d-bc1d-8b2906a25bf3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:26:07,456 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:26:07,471 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:26:07,501 DEBUG awx.main.dispatch task 53b785d7-740a-477d-bc1d-8b2906a25bf3 is finished
2021-05-03 23:26:27,443 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bfe94618-1389-43ac-bc9b-01d0e67de565, queue=awx_private_queue)
[2021-05-03 23:26:27,443: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bfe94618-1389-43ac-bc9b-01d0e67de565, queue=awx_private_queue)
2021-05-03 23:26:27,473 DEBUG awx.main.dispatch delivered bfe94618-1389-43ac-bc9b-01d0e67de565 to worker[208] qsize 0
2021-05-03 23:26:27,480 DEBUG awx.main.dispatch task bfe94618-1389-43ac-bc9b-01d0e67de565 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:26:27,484 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:26:27,509 DEBUG awx.main.scheduler Starting Scheduler
2021-05-03 23:26:27,538 DEBUG awx.main.dispatch task bfe94618-1389-43ac-bc9b-01d0e67de565 is finished
2021-05-03 23:26:37,344 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b72d44a6-7893-4440-9889-bf91f8062d90, queue=awx_private_queue)
[2021-05-03 23:26:37,344: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b72d44a6-7893-4440-9889-bf91f8062d90, queue=awx_private_queue)
2021-05-03 23:26:37,360 DEBUG awx.main.dispatch delivered b72d44a6-7893-4440-9889-bf91f8062d90 to worker[202] qsize 0
2021-05-03 23:26:37,365 DEBUG awx.main.dispatch task b72d44a6-7893-4440-9889-bf91f8062d90 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:26:37,378 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:26:37,383 DEBUG awx.main.tasks Last scheduler run was: 2021-05-03 23:26:07.372885+00:00
2021-05-03 23:26:37,398 DEBUG awx.main.dispatch task b72d44a6-7893-4440-9889-bf91f8062d90 is finished
2021-05-03 23:26:47,440 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(24894d04-6020-4071-a114-f9e42d8d8837, queue=awx_private_queue)
[2021-05-03 23:26:47,440: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(24894d04-6020-4071-a114-f9e42d8d8837, queue=awx_private_queue)
2021-05-03 23:26:47,456 DEBUG awx.main.dispatch delivered 24894d04-6020-4071-a114-f9e42d8d8837 to worker[208] qsize 0
2021-05-03 23:26:47,460 DEBUG awx.main.dispatch task 24894d04-6020-4071-a114-f9e42d8d8837 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:26:47,462 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:26:47,474 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:26:47,497 DEBUG awx.main.dispatch task 24894d04-6020-4071-a114-f9e42d8d8837 is finished
RESULT 2
OKREADY
2021-05-03 23:27:07,351 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ddef0c50-6632-4468-823d-643ca2cc0e19, queue=awx_private_queue)
[2021-05-03 23:27:07,351: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ddef0c50-6632-4468-823d-643ca2cc0e19, queue=awx_private_queue)
2021-05-03 23:27:07,421 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8e51e3f1-5866-494e-bfdc-3e4886621462, queue=dev-awx-0)
[2021-05-03 23:27:07,421: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8e51e3f1-5866-494e-bfdc-3e4886621462, queue=dev-awx-0)
2021-05-03 23:27:07,473 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(80796500-7da6-4a17-9122-016447a0d615, queue=dev-awx-0)
[2021-05-03 23:27:07,473: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(80796500-7da6-4a17-9122-016447a0d615, queue=dev-awx-0)
2021-05-03 23:27:07,506 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3de287a6-5f20-4c71-98d0-701bbb1d115c, queue=awx_private_queue)
[2021-05-03 23:27:07,506: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3de287a6-5f20-4c71-98d0-701bbb1d115c, queue=awx_private_queue)
2021-05-03 23:27:07,420 DEBUG awx.main.dispatch delivered ddef0c50-6632-4468-823d-643ca2cc0e19 to worker[204] qsize 0
2021-05-03 23:27:07,620 DEBUG awx.main.dispatch task ddef0c50-6632-4468-823d-643ca2cc0e19 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:27:07,628 DEBUG awx.main.dispatch delivered 8e51e3f1-5866-494e-bfdc-3e4886621462 to worker[197] qsize 0
2021-05-03 23:27:07,632 DEBUG awx.main.dispatch task 8e51e3f1-5866-494e-bfdc-3e4886621462 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:27:07,637 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:27:07,637 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:27:07,631 DEBUG awx.main.dispatch delivered 80796500-7da6-4a17-9122-016447a0d615 to worker[208] qsize 0
2021-05-03 23:27:07,640 DEBUG awx.main.dispatch delivered 3de287a6-5f20-4c71-98d0-701bbb1d115c to worker[202] qsize 0
2021-05-03 23:27:07,642 DEBUG awx.main.dispatch task 80796500-7da6-4a17-9122-016447a0d615 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:27:07,644 DEBUG awx.main.dispatch task 3de287a6-5f20-4c71-98d0-701bbb1d115c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:27:07,651 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:27:07,643 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:27:07,653 DEBUG awx.main.dispatch task ddef0c50-6632-4468-823d-643ca2cc0e19 is finished
2021-05-03 23:27:07,650 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-03 23:27:07,655 DEBUG awx.main.dispatch task 8e51e3f1-5866-494e-bfdc-3e4886621462 is finished
2021-05-03 23:27:07,661 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140202230842944_3" does not exist
2021-05-03 23:27:07,665 DEBUG awx.main.dispatch task 80796500-7da6-4a17-9122-016447a0d615 is finished
2021-05-03 23:27:07,664 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:27:07,671 DEBUG awx.main.dispatch task 3de287a6-5f20-4c71-98d0-701bbb1d115c is finished
2021-05-03 23:27:27,534 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0999c6bf-f751-45e6-898c-912657037d88, queue=awx_private_queue)
[2021-05-03 23:27:27,534: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0999c6bf-f751-45e6-898c-912657037d88, queue=awx_private_queue)
2021-05-03 23:27:27,560 DEBUG awx.main.dispatch delivered 0999c6bf-f751-45e6-898c-912657037d88 to worker[202] qsize 0
2021-05-03 23:27:27,567 DEBUG awx.main.dispatch task 0999c6bf-f751-45e6-898c-912657037d88 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:27:27,569 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:27:27,582 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:27:27,585 DEBUG awx.main.dispatch task 0999c6bf-f751-45e6-898c-912657037d88 is finished
2021-05-03 23:27:37,352 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1ce9c439-9bc5-49ce-92d2-b7874923bd71, queue=awx_private_queue)
[2021-05-03 23:27:37,352: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1ce9c439-9bc5-49ce-92d2-b7874923bd71, queue=awx_private_queue)
2021-05-03 23:27:37,367 DEBUG awx.main.dispatch delivered 1ce9c439-9bc5-49ce-92d2-b7874923bd71 to worker[204] qsize 0
2021-05-03 23:27:37,373 DEBUG awx.main.dispatch task 1ce9c439-9bc5-49ce-92d2-b7874923bd71 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:27:37,387 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:27:37,392 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:27:37,395 DEBUG awx.main.dispatch task 1ce9c439-9bc5-49ce-92d2-b7874923bd71 is finished
2021-05-03 23:27:47,517 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e9802c73-a986-45eb-b735-cd3198794980, queue=awx_private_queue)
[2021-05-03 23:27:47,517: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e9802c73-a986-45eb-b735-cd3198794980, queue=awx_private_queue)
2021-05-03 23:27:47,543 DEBUG awx.main.dispatch delivered e9802c73-a986-45eb-b735-cd3198794980 to worker[208] qsize 0
2021-05-03 23:27:47,551 DEBUG awx.main.dispatch task e9802c73-a986-45eb-b735-cd3198794980 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:27:47,557 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:27:47,571 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:27:47,574 DEBUG awx.main.dispatch task e9802c73-a986-45eb-b735-cd3198794980 is finished
RESULT 2
OKREADY
2021-05-03 23:28:07,360 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(983fd4f7-5773-4327-ab8a-1168d87d15d6, queue=awx_private_queue)
[2021-05-03 23:28:07,360: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(983fd4f7-5773-4327-ab8a-1168d87d15d6, queue=awx_private_queue)
2021-05-03 23:28:07,421 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(3400785a-24f0-4d38-bf0b-0dcb5b21f7f4, queue=dev-awx-0)
[2021-05-03 23:28:07,421: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(3400785a-24f0-4d38-bf0b-0dcb5b21f7f4, queue=dev-awx-0)
2021-05-03 23:28:07,474 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b920a8f4-916d-45fa-80d8-0f86a6413e2a, queue=dev-awx-0)
[2021-05-03 23:28:07,474: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b920a8f4-916d-45fa-80d8-0f86a6413e2a, queue=dev-awx-0)
2021-05-03 23:28:07,520 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8be0661a-73e8-4a7d-b6fd-809193118d3a, queue=awx_private_queue)
[2021-05-03 23:28:07,520: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8be0661a-73e8-4a7d-b6fd-809193118d3a, queue=awx_private_queue)
2021-05-03 23:28:07,407 DEBUG awx.main.dispatch delivered 983fd4f7-5773-4327-ab8a-1168d87d15d6 to worker[202] qsize 0
2021-05-03 23:28:07,536 DEBUG awx.main.dispatch task 983fd4f7-5773-4327-ab8a-1168d87d15d6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:28:07,562 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:28:07,568 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:28:07,572 DEBUG awx.main.dispatch task 983fd4f7-5773-4327-ab8a-1168d87d15d6 is finished
2021-05-03 23:28:07,573 DEBUG awx.main.dispatch delivered 3400785a-24f0-4d38-bf0b-0dcb5b21f7f4 to worker[204] qsize 0
2021-05-03 23:28:07,789 DEBUG awx.main.dispatch delivered b920a8f4-916d-45fa-80d8-0f86a6413e2a to worker[208] qsize 0
2021-05-03 23:28:07,791 DEBUG awx.main.dispatch delivered 8be0661a-73e8-4a7d-b6fd-809193118d3a to worker[202] qsize 0
2021-05-03 23:28:07,790 DEBUG awx.main.dispatch task 3400785a-24f0-4d38-bf0b-0dcb5b21f7f4 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:28:07,793 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:28:07,793 DEBUG awx.main.dispatch task b920a8f4-916d-45fa-80d8-0f86a6413e2a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:28:07,794 DEBUG awx.main.dispatch task 8be0661a-73e8-4a7d-b6fd-809193118d3a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:28:07,796 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:28:07,805 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:28:07,806 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140202230842944_4" does not exist
2021-05-03 23:28:07,807 DEBUG awx.main.dispatch task 8be0661a-73e8-4a7d-b6fd-809193118d3a is finished
2021-05-03 23:28:07,807 DEBUG awx.main.dispatch task b920a8f4-916d-45fa-80d8-0f86a6413e2a is finished
2021-05-03 23:28:07,810 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-03 23:28:07,812 DEBUG awx.main.dispatch task 3400785a-24f0-4d38-bf0b-0dcb5b21f7f4 is finished
2021-05-03 23:28:27,529 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0d9d3af2-723f-4c2a-af12-2a46d1ad4f3c, queue=awx_private_queue)
[2021-05-03 23:28:27,529: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0d9d3af2-723f-4c2a-af12-2a46d1ad4f3c, queue=awx_private_queue)
2021-05-03 23:28:27,551 DEBUG awx.main.dispatch delivered 0d9d3af2-723f-4c2a-af12-2a46d1ad4f3c to worker[197] qsize 0
2021-05-03 23:28:27,557 DEBUG awx.main.dispatch task 0d9d3af2-723f-4c2a-af12-2a46d1ad4f3c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:28:27,561 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:28:27,575 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:28:27,577 DEBUG awx.main.dispatch task 0d9d3af2-723f-4c2a-af12-2a46d1ad4f3c is finished
2021-05-03 23:28:37,360 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(47c5c7b3-0abe-4158-a48d-44fff5206761, queue=awx_private_queue)
[2021-05-03 23:28:37,360: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(47c5c7b3-0abe-4158-a48d-44fff5206761, queue=awx_private_queue)
2021-05-03 23:28:37,374 DEBUG awx.main.dispatch delivered 47c5c7b3-0abe-4158-a48d-44fff5206761 to worker[202] qsize 0
2021-05-03 23:28:37,379 DEBUG awx.main.dispatch task 47c5c7b3-0abe-4158-a48d-44fff5206761 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:28:37,389 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:28:37,394 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:28:37,397 DEBUG awx.main.dispatch task 47c5c7b3-0abe-4158-a48d-44fff5206761 is finished
2021-05-03 23:28:47,524 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(58be6924-c1d8-4539-b3cd-198372268ca0, queue=awx_private_queue)
[2021-05-03 23:28:47,524: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(58be6924-c1d8-4539-b3cd-198372268ca0, queue=awx_private_queue)
2021-05-03 23:28:47,543 DEBUG awx.main.dispatch delivered 58be6924-c1d8-4539-b3cd-198372268ca0 to worker[197] qsize 0
2021-05-03 23:28:47,547 DEBUG awx.main.dispatch task 58be6924-c1d8-4539-b3cd-198372268ca0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:28:47,549 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:28:47,558 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:28:47,562 DEBUG awx.main.dispatch task 58be6924-c1d8-4539-b3cd-198372268ca0 is finished
RESULT 2
OKREADY
2021-05-03 23:29:07,368 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(38d68af5-760e-4630-bce9-0026fe480d4a, queue=awx_private_queue)
[2021-05-03 23:29:07,368: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(38d68af5-760e-4630-bce9-0026fe480d4a, queue=awx_private_queue)
2021-05-03 23:29:07,421 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c700f2b7-0fbc-4a3a-8043-b262f927e2ab, queue=dev-awx-0)
[2021-05-03 23:29:07,421: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c700f2b7-0fbc-4a3a-8043-b262f927e2ab, queue=dev-awx-0)
2021-05-03 23:29:07,474 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(e0b9834e-1324-45df-9ca4-0afe55c1aad5, queue=dev-awx-0)
[2021-05-03 23:29:07,474: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(e0b9834e-1324-45df-9ca4-0afe55c1aad5, queue=dev-awx-0)
2021-05-03 23:29:07,523 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cf4ea5bc-3288-4e81-8630-d261d721aeb2, queue=awx_private_queue)
[2021-05-03 23:29:07,523: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cf4ea5bc-3288-4e81-8630-d261d721aeb2, queue=awx_private_queue)
2021-05-03 23:29:07,401 DEBUG awx.main.dispatch delivered 38d68af5-760e-4630-bce9-0026fe480d4a to worker[208] qsize 0
2021-05-03 23:29:08,500 DEBUG awx.main.dispatch task 38d68af5-760e-4630-bce9-0026fe480d4a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:29:08,591 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:29:08,595 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:29:08,600 DEBUG awx.main.dispatch task 38d68af5-760e-4630-bce9-0026fe480d4a is finished
2021-05-03 23:29:08,516 DEBUG awx.main.dispatch delivered c700f2b7-0fbc-4a3a-8043-b262f927e2ab to worker[204] qsize 0
2021-05-03 23:29:08,696 DEBUG awx.main.dispatch delivered e0b9834e-1324-45df-9ca4-0afe55c1aad5 to worker[197] qsize 0
2021-05-03 23:29:08,697 DEBUG awx.main.dispatch delivered cf4ea5bc-3288-4e81-8630-d261d721aeb2 to worker[208] qsize 0
2021-05-03 23:29:08,703 DEBUG awx.main.dispatch task cf4ea5bc-3288-4e81-8630-d261d721aeb2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:29:08,705 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:29:08,700 DEBUG awx.main.dispatch task c700f2b7-0fbc-4a3a-8043-b262f927e2ab starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:29:08,707 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:29:08,706 DEBUG awx.main.dispatch task e0b9834e-1324-45df-9ca4-0afe55c1aad5 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:29:08,713 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:29:08,715 DEBUG awx.main.dispatch task cf4ea5bc-3288-4e81-8630-d261d721aeb2 is finished
2021-05-03 23:29:08,722 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-03 23:29:08,724 DEBUG awx.main.dispatch task c700f2b7-0fbc-4a3a-8043-b262f927e2ab is finished
2021-05-03 23:29:08,724 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140202230842944_4" does not exist
2021-05-03 23:29:08,727 DEBUG awx.main.dispatch task e0b9834e-1324-45df-9ca4-0afe55c1aad5 is finished
2021-05-03 23:29:27,537 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(602f8249-f385-4b9f-93ac-eea6c75f6321, queue=awx_private_queue)
[2021-05-03 23:29:27,537: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(602f8249-f385-4b9f-93ac-eea6c75f6321, queue=awx_private_queue)
2021-05-03 23:29:27,557 DEBUG awx.main.dispatch delivered 602f8249-f385-4b9f-93ac-eea6c75f6321 to worker[204] qsize 0
2021-05-03 23:29:27,560 DEBUG awx.main.dispatch task 602f8249-f385-4b9f-93ac-eea6c75f6321 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:29:27,562 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:29:27,572 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:29:27,575 DEBUG awx.main.dispatch task 602f8249-f385-4b9f-93ac-eea6c75f6321 is finished
2021-05-03 23:29:37,375 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ab06347c-fcbc-44cb-9460-00d00fb48f4d, queue=awx_private_queue)
[2021-05-03 23:29:37,375: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ab06347c-fcbc-44cb-9460-00d00fb48f4d, queue=awx_private_queue)
2021-05-03 23:29:37,408 DEBUG awx.main.dispatch delivered ab06347c-fcbc-44cb-9460-00d00fb48f4d to worker[197] qsize 0
2021-05-03 23:29:37,415 DEBUG awx.main.dispatch task ab06347c-fcbc-44cb-9460-00d00fb48f4d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:29:37,429 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:29:37,434 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:29:37,437 DEBUG awx.main.dispatch task ab06347c-fcbc-44cb-9460-00d00fb48f4d is finished
2021-05-03 23:29:47,535 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(589b6ce0-033a-4393-99ec-7154accd2eab, queue=awx_private_queue)
[2021-05-03 23:29:47,535: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(589b6ce0-033a-4393-99ec-7154accd2eab, queue=awx_private_queue)
2021-05-03 23:29:47,554 DEBUG awx.main.dispatch delivered 589b6ce0-033a-4393-99ec-7154accd2eab to worker[204] qsize 0
2021-05-03 23:29:47,561 DEBUG awx.main.dispatch task 589b6ce0-033a-4393-99ec-7154accd2eab starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:29:47,565 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:29:47,581 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:29:47,583 DEBUG awx.main.dispatch task 589b6ce0-033a-4393-99ec-7154accd2eab is finished
RESULT 2
OKREADY
2021-05-03 23:30:07,377 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b8d8d8a9-4ef9-4c39-8e04-687b57bde01f, queue=awx_private_queue)
[2021-05-03 23:30:07,377: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b8d8d8a9-4ef9-4c39-8e04-687b57bde01f, queue=awx_private_queue)
2021-05-03 23:30:07,404 DEBUG awx.main.dispatch delivered b8d8d8a9-4ef9-4c39-8e04-687b57bde01f to worker[202] qsize 0
2021-05-03 23:30:07,412 DEBUG awx.main.dispatch task b8d8d8a9-4ef9-4c39-8e04-687b57bde01f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:30:07,423 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(0f183337-03fa-49d7-bad0-814d3ac0b0ee, queue=dev-awx-0)
[2021-05-03 23:30:07,423: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(0f183337-03fa-49d7-bad0-814d3ac0b0ee, queue=dev-awx-0)
2021-05-03 23:30:07,438 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:30:07,446 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:30:07,450 DEBUG awx.main.dispatch task b8d8d8a9-4ef9-4c39-8e04-687b57bde01f is finished
2021-05-03 23:30:07,472 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2530fce8-c541-4d6a-8482-bed4fcd9168a, queue=dev-awx-0)
[2021-05-03 23:30:07,472: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2530fce8-c541-4d6a-8482-bed4fcd9168a, queue=dev-awx-0)
2021-05-03 23:30:07,474 DEBUG awx.main.dispatch delivered 0f183337-03fa-49d7-bad0-814d3ac0b0ee to worker[197] qsize 0
2021-05-03 23:30:07,477 DEBUG awx.main.dispatch task 0f183337-03fa-49d7-bad0-814d3ac0b0ee starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:30:07,480 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:30:07,489 DEBUG awx.main.dispatch delivered 2530fce8-c541-4d6a-8482-bed4fcd9168a to worker[204] qsize 0
2021-05-03 23:30:07,492 DEBUG awx.main.dispatch task 2530fce8-c541-4d6a-8482-bed4fcd9168a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:30:07,492 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-03 23:30:07,495 DEBUG awx.main.dispatch task 0f183337-03fa-49d7-bad0-814d3ac0b0ee is finished
2021-05-03 23:30:07,506 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140202230842944_4" does not exist
2021-05-03 23:30:07,508 DEBUG awx.main.dispatch task 2530fce8-c541-4d6a-8482-bed4fcd9168a is finished
2021-05-03 23:30:07,534 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6690234d-d102-4bbb-8fd3-dbcad523d1bb, queue=awx_private_queue)
[2021-05-03 23:30:07,534: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6690234d-d102-4bbb-8fd3-dbcad523d1bb, queue=awx_private_queue)
2021-05-03 23:30:07,559 DEBUG awx.main.dispatch delivered 6690234d-d102-4bbb-8fd3-dbcad523d1bb to worker[208] qsize 0
2021-05-03 23:30:07,566 DEBUG awx.main.dispatch task 6690234d-d102-4bbb-8fd3-dbcad523d1bb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:30:07,571 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:30:07,588 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:30:07,592 DEBUG awx.main.dispatch task 6690234d-d102-4bbb-8fd3-dbcad523d1bb is finished
2021-05-03 23:30:27,543 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ec395a8d-3df5-4595-b268-469c99bf210d, queue=awx_private_queue)
[2021-05-03 23:30:27,543: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ec395a8d-3df5-4595-b268-469c99bf210d, queue=awx_private_queue)
2021-05-03 23:30:27,587 DEBUG awx.main.dispatch delivered ec395a8d-3df5-4595-b268-469c99bf210d to worker[202] qsize 0
2021-05-03 23:30:27,771 DEBUG awx.main.dispatch task ec395a8d-3df5-4595-b268-469c99bf210d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:30:27,777 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:30:27,788 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:30:27,790 DEBUG awx.main.dispatch task ec395a8d-3df5-4595-b268-469c99bf210d is finished
2021-05-03 23:30:37,381 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d0c5fd74-baed-41ca-a84c-d4d22ea04f26, queue=awx_private_queue)
[2021-05-03 23:30:37,381: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d0c5fd74-baed-41ca-a84c-d4d22ea04f26, queue=awx_private_queue)
2021-05-03 23:30:37,403 DEBUG awx.main.dispatch delivered d0c5fd74-baed-41ca-a84c-d4d22ea04f26 to worker[208] qsize 0
2021-05-03 23:30:37,409 DEBUG awx.main.dispatch task d0c5fd74-baed-41ca-a84c-d4d22ea04f26 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:30:37,423 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:30:37,428 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:30:37,430 DEBUG awx.main.dispatch task d0c5fd74-baed-41ca-a84c-d4d22ea04f26 is finished
2021-05-03 23:30:47,545 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b77b15fc-0d9c-447d-8c2a-3a8168922fe7, queue=awx_private_queue)
[2021-05-03 23:30:47,545: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b77b15fc-0d9c-447d-8c2a-3a8168922fe7, queue=awx_private_queue)
2021-05-03 23:30:47,567 DEBUG awx.main.dispatch delivered b77b15fc-0d9c-447d-8c2a-3a8168922fe7 to worker[208] qsize 0
2021-05-03 23:30:47,572 DEBUG awx.main.dispatch task b77b15fc-0d9c-447d-8c2a-3a8168922fe7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:30:47,573 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:30:47,584 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:30:47,586 DEBUG awx.main.dispatch task b77b15fc-0d9c-447d-8c2a-3a8168922fe7 is finished
RESULT 2
OKREADY
2021-05-03 23:31:07,386 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(eedd2b71-b40e-47a8-a7c0-fbd82c962db9, queue=awx_private_queue)
[2021-05-03 23:31:07,386: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(eedd2b71-b40e-47a8-a7c0-fbd82c962db9, queue=awx_private_queue)
2021-05-03 23:31:07,405 DEBUG awx.main.dispatch delivered eedd2b71-b40e-47a8-a7c0-fbd82c962db9 to worker[204] qsize 0
2021-05-03 23:31:07,412 DEBUG awx.main.dispatch task eedd2b71-b40e-47a8-a7c0-fbd82c962db9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:31:07,420 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8b3c2f54-9663-47ba-851e-d074a2a4ee6e, queue=dev-awx-0)
[2021-05-03 23:31:07,420: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8b3c2f54-9663-47ba-851e-d074a2a4ee6e, queue=dev-awx-0)
2021-05-03 23:31:07,429 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:31:07,435 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:31:07,438 DEBUG awx.main.dispatch task eedd2b71-b40e-47a8-a7c0-fbd82c962db9 is finished
2021-05-03 23:31:07,457 DEBUG awx.main.dispatch delivered 8b3c2f54-9663-47ba-851e-d074a2a4ee6e to worker[202] qsize 0
2021-05-03 23:31:07,461 DEBUG awx.main.dispatch task 8b3c2f54-9663-47ba-851e-d074a2a4ee6e starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:31:07,465 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:31:07,472 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b9f2a9da-85e3-4f83-bad6-4c1663bd5b5b, queue=dev-awx-0)
[2021-05-03 23:31:07,472: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b9f2a9da-85e3-4f83-bad6-4c1663bd5b5b, queue=dev-awx-0)
2021-05-03 23:31:07,482 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-03 23:31:07,485 DEBUG awx.main.dispatch task 8b3c2f54-9663-47ba-851e-d074a2a4ee6e is finished
2021-05-03 23:31:07,486 DEBUG awx.main.dispatch delivered b9f2a9da-85e3-4f83-bad6-4c1663bd5b5b to worker[197] qsize 0
2021-05-03 23:31:07,493 DEBUG awx.main.dispatch task b9f2a9da-85e3-4f83-bad6-4c1663bd5b5b starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:31:07,513 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140202230842944_5" does not exist
2021-05-03 23:31:07,515 DEBUG awx.main.dispatch task b9f2a9da-85e3-4f83-bad6-4c1663bd5b5b is finished
2021-05-03 23:31:07,544 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e69054f3-fe67-4f88-94e9-7aafab825ea4, queue=awx_private_queue)
[2021-05-03 23:31:07,544: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e69054f3-fe67-4f88-94e9-7aafab825ea4, queue=awx_private_queue)
2021-05-03 23:31:07,564 DEBUG awx.main.dispatch delivered e69054f3-fe67-4f88-94e9-7aafab825ea4 to worker[202] qsize 0
2021-05-03 23:31:07,568 DEBUG awx.main.dispatch task e69054f3-fe67-4f88-94e9-7aafab825ea4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:31:07,570 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:31:07,578 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:31:07,579 DEBUG awx.main.dispatch task e69054f3-fe67-4f88-94e9-7aafab825ea4 is finished
2021-05-03 23:31:27,554 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(04a8744c-ce3c-4c8d-9197-4ace462c5f66, queue=awx_private_queue)
[2021-05-03 23:31:27,554: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(04a8744c-ce3c-4c8d-9197-4ace462c5f66, queue=awx_private_queue)
2021-05-03 23:31:27,600 DEBUG awx.main.dispatch delivered 04a8744c-ce3c-4c8d-9197-4ace462c5f66 to worker[197] qsize 0
2021-05-03 23:31:27,797 DEBUG awx.main.dispatch task 04a8744c-ce3c-4c8d-9197-4ace462c5f66 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:31:27,811 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:31:27,831 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:31:27,835 DEBUG awx.main.dispatch task 04a8744c-ce3c-4c8d-9197-4ace462c5f66 is finished
2021-05-03 23:31:37,388 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d99e7115-8ebe-4d97-984c-65a5265363fc, queue=awx_private_queue)
[2021-05-03 23:31:37,388: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d99e7115-8ebe-4d97-984c-65a5265363fc, queue=awx_private_queue)
2021-05-03 23:31:37,412 DEBUG awx.main.dispatch delivered d99e7115-8ebe-4d97-984c-65a5265363fc to worker[202] qsize 0
2021-05-03 23:31:37,418 DEBUG awx.main.dispatch task d99e7115-8ebe-4d97-984c-65a5265363fc starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:31:37,432 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:31:37,436 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:31:37,442 DEBUG awx.main.dispatch task d99e7115-8ebe-4d97-984c-65a5265363fc is finished
2021-05-03 23:31:47,555 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(db1205c3-d79b-4adc-b8eb-7e88fc2669b0, queue=awx_private_queue)
[2021-05-03 23:31:47,555: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(db1205c3-d79b-4adc-b8eb-7e88fc2669b0, queue=awx_private_queue)
2021-05-03 23:31:47,569 DEBUG awx.main.dispatch delivered db1205c3-d79b-4adc-b8eb-7e88fc2669b0 to worker[204] qsize 0
2021-05-03 23:31:47,574 DEBUG awx.main.dispatch task db1205c3-d79b-4adc-b8eb-7e88fc2669b0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:31:47,576 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:31:47,586 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:31:47,589 DEBUG awx.main.dispatch task db1205c3-d79b-4adc-b8eb-7e88fc2669b0 is finished
RESULT 2
OKREADY
2021-05-03 23:32:07,404 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3091efcc-3589-4189-8f16-dbad4a10c16f, queue=awx_private_queue)
[2021-05-03 23:32:07,404: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3091efcc-3589-4189-8f16-dbad4a10c16f, queue=awx_private_queue)
2021-05-03 23:32:07,429 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(324779ef-8520-4e38-859c-58379dc65da3, queue=dev-awx-0)
[2021-05-03 23:32:07,429: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(324779ef-8520-4e38-859c-58379dc65da3, queue=dev-awx-0)
2021-05-03 23:32:07,431 DEBUG awx.main.dispatch delivered 3091efcc-3589-4189-8f16-dbad4a10c16f to worker[197] qsize 0
2021-05-03 23:32:07,442 DEBUG awx.main.dispatch task 3091efcc-3589-4189-8f16-dbad4a10c16f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:32:07,460 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:32:07,468 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:32:07,471 DEBUG awx.main.dispatch task 3091efcc-3589-4189-8f16-dbad4a10c16f is finished
2021-05-03 23:32:07,472 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(24d3ee43-9703-4be7-ad63-96c1c272ad1c, queue=dev-awx-0)
[2021-05-03 23:32:07,472: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(24d3ee43-9703-4be7-ad63-96c1c272ad1c, queue=dev-awx-0)
2021-05-03 23:32:07,479 DEBUG awx.main.dispatch delivered 324779ef-8520-4e38-859c-58379dc65da3 to worker[204] qsize 0
2021-05-03 23:32:07,484 DEBUG awx.main.dispatch task 324779ef-8520-4e38-859c-58379dc65da3 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:32:07,490 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:32:07,501 DEBUG awx.main.dispatch delivered 24d3ee43-9703-4be7-ad63-96c1c272ad1c to worker[202] qsize 0
2021-05-03 23:32:07,518 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-03 23:32:07,520 DEBUG awx.main.dispatch task 324779ef-8520-4e38-859c-58379dc65da3 is finished
2021-05-03 23:32:07,511 DEBUG awx.main.dispatch task 24d3ee43-9703-4be7-ad63-96c1c272ad1c starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:32:07,540 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140202230842944_4" does not exist
2021-05-03 23:32:07,544 DEBUG awx.main.dispatch task 24d3ee43-9703-4be7-ad63-96c1c272ad1c is finished
2021-05-03 23:32:07,555 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(97484944-66c8-4a97-9b24-e815991bbcb5, queue=awx_private_queue)
[2021-05-03 23:32:07,555: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(97484944-66c8-4a97-9b24-e815991bbcb5, queue=awx_private_queue)
2021-05-03 23:32:07,580 DEBUG awx.main.dispatch delivered 97484944-66c8-4a97-9b24-e815991bbcb5 to worker[197] qsize 0
2021-05-03 23:32:07,582 DEBUG awx.main.dispatch task 97484944-66c8-4a97-9b24-e815991bbcb5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:32:07,584 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:32:07,597 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:32:07,599 DEBUG awx.main.dispatch task 97484944-66c8-4a97-9b24-e815991bbcb5 is finished
2021-05-03 23:32:27,561 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ba5d7178-b08e-4fa5-ade6-87cdab628420, queue=awx_private_queue)
[2021-05-03 23:32:27,561: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ba5d7178-b08e-4fa5-ade6-87cdab628420, queue=awx_private_queue)
2021-05-03 23:32:27,597 DEBUG awx.main.dispatch delivered ba5d7178-b08e-4fa5-ade6-87cdab628420 to worker[202] qsize 0
2021-05-03 23:32:27,609 DEBUG awx.main.dispatch task ba5d7178-b08e-4fa5-ade6-87cdab628420 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:32:27,615 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:32:27,625 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:32:27,627 DEBUG awx.main.dispatch task ba5d7178-b08e-4fa5-ade6-87cdab628420 is finished
2021-05-03 23:32:37,400 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(494958b8-a49e-4b2d-8b15-b4045ba23228, queue=awx_private_queue)
[2021-05-03 23:32:37,400: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(494958b8-a49e-4b2d-8b15-b4045ba23228, queue=awx_private_queue)
2021-05-03 23:32:37,436 DEBUG awx.main.dispatch delivered 494958b8-a49e-4b2d-8b15-b4045ba23228 to worker[197] qsize 0
2021-05-03 23:32:37,641 DEBUG awx.main.dispatch task 494958b8-a49e-4b2d-8b15-b4045ba23228 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:32:37,658 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:32:37,663 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:32:37,665 DEBUG awx.main.dispatch task 494958b8-a49e-4b2d-8b15-b4045ba23228 is finished
2021-05-03 23:32:47,563 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7695bc38-7b89-4a21-828e-409fee74938e, queue=awx_private_queue)
[2021-05-03 23:32:47,563: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7695bc38-7b89-4a21-828e-409fee74938e, queue=awx_private_queue)
2021-05-03 23:32:47,585 DEBUG awx.main.dispatch delivered 7695bc38-7b89-4a21-828e-409fee74938e to worker[204] qsize 0
2021-05-03 23:32:47,593 DEBUG awx.main.dispatch task 7695bc38-7b89-4a21-828e-409fee74938e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:32:47,598 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:32:47,612 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:32:47,616 DEBUG awx.main.dispatch task 7695bc38-7b89-4a21-828e-409fee74938e is finished
RESULT 2
OKREADY
2021-05-03 23:33:07,405 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c8d97eb1-0bb6-4a59-8306-991e8916cdca, queue=awx_private_queue)
[2021-05-03 23:33:07,405: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c8d97eb1-0bb6-4a59-8306-991e8916cdca, queue=awx_private_queue)
2021-05-03 23:33:07,436 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(540a2a45-d065-4a25-9389-6d1ccb561eae, queue=dev-awx-0)
[2021-05-03 23:33:07,436: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(540a2a45-d065-4a25-9389-6d1ccb561eae, queue=dev-awx-0)
2021-05-03 23:33:07,434 DEBUG awx.main.dispatch delivered c8d97eb1-0bb6-4a59-8306-991e8916cdca to worker[197] qsize 0
2021-05-03 23:33:07,444 DEBUG awx.main.dispatch task c8d97eb1-0bb6-4a59-8306-991e8916cdca starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:33:07,462 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:33:07,465 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:33:07,467 DEBUG awx.main.dispatch task c8d97eb1-0bb6-4a59-8306-991e8916cdca is finished
2021-05-03 23:33:07,473 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5fd1e044-5e58-4c9d-a949-d7e4c074e826, queue=dev-awx-0)
[2021-05-03 23:33:07,473: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5fd1e044-5e58-4c9d-a949-d7e4c074e826, queue=dev-awx-0)
2021-05-03 23:33:07,499 DEBUG awx.main.dispatch delivered 540a2a45-d065-4a25-9389-6d1ccb561eae to worker[208] qsize 0
2021-05-03 23:33:07,504 DEBUG awx.main.dispatch delivered 5fd1e044-5e58-4c9d-a949-d7e4c074e826 to worker[204] qsize 0
2021-05-03 23:33:07,508 DEBUG awx.main.dispatch task 5fd1e044-5e58-4c9d-a949-d7e4c074e826 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:33:07,507 DEBUG awx.main.dispatch task 540a2a45-d065-4a25-9389-6d1ccb561eae starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:33:07,516 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:33:07,521 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140202230842944_5" does not exist
2021-05-03 23:33:07,523 DEBUG awx.main.dispatch task 5fd1e044-5e58-4c9d-a949-d7e4c074e826 is finished
2021-05-03 23:33:07,529 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-03 23:33:07,531 DEBUG awx.main.dispatch task 540a2a45-d065-4a25-9389-6d1ccb561eae is finished
2021-05-03 23:33:07,562 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ad612653-7078-4344-a05b-0974ff29b785, queue=awx_private_queue)
[2021-05-03 23:33:07,562: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ad612653-7078-4344-a05b-0974ff29b785, queue=awx_private_queue)
2021-05-03 23:33:07,579 DEBUG awx.main.dispatch delivered ad612653-7078-4344-a05b-0974ff29b785 to worker[204] qsize 0
2021-05-03 23:33:07,588 DEBUG awx.main.dispatch task ad612653-7078-4344-a05b-0974ff29b785 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:33:07,591 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:33:07,600 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:33:07,603 DEBUG awx.main.dispatch task ad612653-7078-4344-a05b-0974ff29b785 is finished
2021-05-03 23:33:27,571 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b49065ca-41cb-425b-850c-e8a0d3514ece, queue=awx_private_queue)
[2021-05-03 23:33:27,571: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b49065ca-41cb-425b-850c-e8a0d3514ece, queue=awx_private_queue)
2021-05-03 23:33:27,586 DEBUG awx.main.dispatch delivered b49065ca-41cb-425b-850c-e8a0d3514ece to worker[204] qsize 0
2021-05-03 23:33:27,590 DEBUG awx.main.dispatch task b49065ca-41cb-425b-850c-e8a0d3514ece starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:33:27,592 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:33:27,607 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:33:27,609 DEBUG awx.main.dispatch task b49065ca-41cb-425b-850c-e8a0d3514ece is finished
2021-05-03 23:33:37,404 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e14c5df2-27f0-496a-9b3d-41bcf96446a1, queue=awx_private_queue)
[2021-05-03 23:33:37,404: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e14c5df2-27f0-496a-9b3d-41bcf96446a1, queue=awx_private_queue)
2021-05-03 23:33:37,444 DEBUG awx.main.dispatch delivered e14c5df2-27f0-496a-9b3d-41bcf96446a1 to worker[202] qsize 0
2021-05-03 23:33:37,634 DEBUG awx.main.dispatch task e14c5df2-27f0-496a-9b3d-41bcf96446a1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:33:37,650 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:33:37,655 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:33:37,658 DEBUG awx.main.dispatch task e14c5df2-27f0-496a-9b3d-41bcf96446a1 is finished
2021-05-03 23:33:47,583 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(831aca6c-0005-4330-8d8e-e13b18fe95c9, queue=awx_private_queue)
[2021-05-03 23:33:47,583: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(831aca6c-0005-4330-8d8e-e13b18fe95c9, queue=awx_private_queue)
2021-05-03 23:33:47,611 DEBUG awx.main.dispatch delivered 831aca6c-0005-4330-8d8e-e13b18fe95c9 to worker[202] qsize 0
2021-05-03 23:33:47,619 DEBUG awx.main.dispatch task 831aca6c-0005-4330-8d8e-e13b18fe95c9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:33:47,621 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:33:47,633 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:33:47,636 DEBUG awx.main.dispatch task 831aca6c-0005-4330-8d8e-e13b18fe95c9 is finished
RESULT 2
OKREADY
2021-05-03 23:34:07,411 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b9b747ca-2ff1-4650-a01d-d80c7d2b6246, queue=awx_private_queue)
[2021-05-03 23:34:07,411: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b9b747ca-2ff1-4650-a01d-d80c7d2b6246, queue=awx_private_queue)
2021-05-03 23:34:07,428 DEBUG awx.main.dispatch delivered b9b747ca-2ff1-4650-a01d-d80c7d2b6246 to worker[202] qsize 0
2021-05-03 23:34:07,434 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a37a84dc-cb35-4766-b4ec-41debd19b1d9, queue=dev-awx-0)
[2021-05-03 23:34:07,434: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a37a84dc-cb35-4766-b4ec-41debd19b1d9, queue=dev-awx-0)
2021-05-03 23:34:07,434 DEBUG awx.main.dispatch task b9b747ca-2ff1-4650-a01d-d80c7d2b6246 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:34:07,447 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:34:07,452 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:34:07,454 DEBUG awx.main.dispatch task b9b747ca-2ff1-4650-a01d-d80c7d2b6246 is finished
2021-05-03 23:34:07,472 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d12e91ed-3bd2-46ef-a485-c81296879cb8, queue=dev-awx-0)
[2021-05-03 23:34:07,472: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d12e91ed-3bd2-46ef-a485-c81296879cb8, queue=dev-awx-0)
2021-05-03 23:34:07,538 DEBUG awx.main.dispatch delivered a37a84dc-cb35-4766-b4ec-41debd19b1d9 to worker[197] qsize 0
2021-05-03 23:34:07,544 DEBUG awx.main.dispatch task a37a84dc-cb35-4766-b4ec-41debd19b1d9 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:34:07,543 DEBUG awx.main.dispatch delivered d12e91ed-3bd2-46ef-a485-c81296879cb8 to worker[204] qsize 0
2021-05-03 23:34:07,549 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:34:07,554 DEBUG awx.main.dispatch task d12e91ed-3bd2-46ef-a485-c81296879cb8 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:34:07,564 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-03 23:34:07,569 DEBUG awx.main.dispatch task a37a84dc-cb35-4766-b4ec-41debd19b1d9 is finished
2021-05-03 23:34:07,572 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(04873f26-0c88-4e9c-a1e8-2a41ef3d4c61, queue=awx_private_queue)
[2021-05-03 23:34:07,572: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(04873f26-0c88-4e9c-a1e8-2a41ef3d4c61, queue=awx_private_queue)
2021-05-03 23:34:07,572 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140202230842944_6" does not exist
2021-05-03 23:34:07,574 DEBUG awx.main.dispatch task d12e91ed-3bd2-46ef-a485-c81296879cb8 is finished
2021-05-03 23:34:07,585 DEBUG awx.main.dispatch delivered 04873f26-0c88-4e9c-a1e8-2a41ef3d4c61 to worker[202] qsize 0
2021-05-03 23:34:07,589 DEBUG awx.main.dispatch task 04873f26-0c88-4e9c-a1e8-2a41ef3d4c61 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:34:07,590 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:34:07,601 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:34:07,603 DEBUG awx.main.dispatch task 04873f26-0c88-4e9c-a1e8-2a41ef3d4c61 is finished
2021-05-03 23:34:27,582 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d80960c6-e70a-4982-9bb6-27eb0175c68c, queue=awx_private_queue)
[2021-05-03 23:34:27,582: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d80960c6-e70a-4982-9bb6-27eb0175c68c, queue=awx_private_queue)
2021-05-03 23:34:27,596 DEBUG awx.main.dispatch delivered d80960c6-e70a-4982-9bb6-27eb0175c68c to worker[202] qsize 0
2021-05-03 23:34:27,603 DEBUG awx.main.dispatch task d80960c6-e70a-4982-9bb6-27eb0175c68c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:34:27,606 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:34:27,618 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:34:27,621 DEBUG awx.main.dispatch task d80960c6-e70a-4982-9bb6-27eb0175c68c is finished
2021-05-03 23:34:37,411 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e4d2a0ec-780c-441e-8960-a8959b50e853, queue=awx_private_queue)
[2021-05-03 23:34:37,411: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e4d2a0ec-780c-441e-8960-a8959b50e853, queue=awx_private_queue)
2021-05-03 23:34:37,446 DEBUG awx.main.dispatch delivered e4d2a0ec-780c-441e-8960-a8959b50e853 to worker[204] qsize 0
2021-05-03 23:34:39,671 DEBUG awx.main.dispatch task e4d2a0ec-780c-441e-8960-a8959b50e853 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:34:39,681 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:34:39,685 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:34:39,688 DEBUG awx.main.dispatch task e4d2a0ec-780c-441e-8960-a8959b50e853 is finished
2021-05-03 23:34:47,587 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(72ebf0ee-9a98-4d7f-8974-7acd5a7d7011, queue=awx_private_queue)
[2021-05-03 23:34:47,587: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(72ebf0ee-9a98-4d7f-8974-7acd5a7d7011, queue=awx_private_queue)
2021-05-03 23:34:47,603 DEBUG awx.main.dispatch delivered 72ebf0ee-9a98-4d7f-8974-7acd5a7d7011 to worker[208] qsize 0
2021-05-03 23:34:47,613 DEBUG awx.main.dispatch task 72ebf0ee-9a98-4d7f-8974-7acd5a7d7011 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:34:47,618 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:34:47,643 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:34:47,645 DEBUG awx.main.dispatch task 72ebf0ee-9a98-4d7f-8974-7acd5a7d7011 is finished
RESULT 2
OKREADY
2021-05-03 23:35:07,425 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2deb2d76-d22f-48c8-871d-e766d086137f, queue=awx_private_queue)
[2021-05-03 23:35:07,425: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2deb2d76-d22f-48c8-871d-e766d086137f, queue=awx_private_queue)
2021-05-03 23:35:07,467 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f2999c83-179b-48f7-a139-bd6d0970973e, queue=dev-awx-0)
[2021-05-03 23:35:07,467: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f2999c83-179b-48f7-a139-bd6d0970973e, queue=dev-awx-0)
2021-05-03 23:35:07,467 DEBUG awx.main.dispatch delivered 2deb2d76-d22f-48c8-871d-e766d086137f to worker[202] qsize 0
2021-05-03 23:35:07,477 DEBUG awx.main.dispatch task 2deb2d76-d22f-48c8-871d-e766d086137f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:35:07,483 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(4daf5d1f-b99c-4480-9b82-1f57b1719ecd, queue=dev-awx-0)
[2021-05-03 23:35:07,483: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(4daf5d1f-b99c-4480-9b82-1f57b1719ecd, queue=dev-awx-0)
2021-05-03 23:35:07,502 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:35:07,509 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:35:07,512 DEBUG awx.main.dispatch task 2deb2d76-d22f-48c8-871d-e766d086137f is finished
2021-05-03 23:35:07,582 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a670c8fc-38e3-46f2-bcda-8f79cb0f84fe, queue=awx_private_queue)
[2021-05-03 23:35:07,582: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a670c8fc-38e3-46f2-bcda-8f79cb0f84fe, queue=awx_private_queue)
2021-05-03 23:35:07,603 DEBUG awx.main.dispatch delivered f2999c83-179b-48f7-a139-bd6d0970973e to worker[202] qsize 0
2021-05-03 23:35:07,606 DEBUG awx.main.dispatch delivered 4daf5d1f-b99c-4480-9b82-1f57b1719ecd to worker[208] qsize 0
2021-05-03 23:35:07,607 DEBUG awx.main.dispatch task f2999c83-179b-48f7-a139-bd6d0970973e starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:35:07,609 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:35:07,610 DEBUG awx.main.dispatch delivered a670c8fc-38e3-46f2-bcda-8f79cb0f84fe to worker[204] qsize 0
2021-05-03 23:35:07,614 DEBUG awx.main.dispatch task 4daf5d1f-b99c-4480-9b82-1f57b1719ecd starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:35:07,616 DEBUG awx.main.dispatch task a670c8fc-38e3-46f2-bcda-8f79cb0f84fe starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:35:07,620 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:35:07,638 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-03 23:35:07,640 DEBUG awx.main.dispatch task f2999c83-179b-48f7-a139-bd6d0970973e is finished
2021-05-03 23:35:07,648 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:35:07,650 DEBUG awx.main.dispatch task a670c8fc-38e3-46f2-bcda-8f79cb0f84fe is finished
2021-05-03 23:35:07,652 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140202230842944_5" does not exist
2021-05-03 23:35:07,656 DEBUG awx.main.dispatch task 4daf5d1f-b99c-4480-9b82-1f57b1719ecd is finished
2021-05-03 23:35:27,594 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b874126d-176c-43ff-a303-0909600ee18b, queue=awx_private_queue)
[2021-05-03 23:35:27,594: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b874126d-176c-43ff-a303-0909600ee18b, queue=awx_private_queue)
2021-05-03 23:35:27,613 DEBUG awx.main.dispatch delivered b874126d-176c-43ff-a303-0909600ee18b to worker[197] qsize 0
2021-05-03 23:35:27,622 DEBUG awx.main.dispatch task b874126d-176c-43ff-a303-0909600ee18b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:35:27,628 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:35:27,649 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:35:27,651 DEBUG awx.main.dispatch task b874126d-176c-43ff-a303-0909600ee18b is finished
2021-05-03 23:35:37,421 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2deb7012-b698-4ffa-9259-891f7a22578f, queue=awx_private_queue)
[2021-05-03 23:35:37,421: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2deb7012-b698-4ffa-9259-891f7a22578f, queue=awx_private_queue)
2021-05-03 23:35:37,435 DEBUG awx.main.dispatch delivered 2deb7012-b698-4ffa-9259-891f7a22578f to worker[202] qsize 0
2021-05-03 23:35:37,442 DEBUG awx.main.dispatch task 2deb7012-b698-4ffa-9259-891f7a22578f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:35:37,456 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:35:37,460 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:35:37,463 DEBUG awx.main.dispatch task 2deb7012-b698-4ffa-9259-891f7a22578f is finished
2021-05-03 23:35:47,594 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7818af2b-1172-4a4f-8654-4af8494abb6a, queue=awx_private_queue)
[2021-05-03 23:35:47,594: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7818af2b-1172-4a4f-8654-4af8494abb6a, queue=awx_private_queue)
2021-05-03 23:35:47,632 DEBUG awx.main.dispatch delivered 7818af2b-1172-4a4f-8654-4af8494abb6a to worker[208] qsize 0
2021-05-03 23:35:52,123 DEBUG awx.main.dispatch task 7818af2b-1172-4a4f-8654-4af8494abb6a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:35:52,129 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:35:52,142 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:35:52,145 DEBUG awx.main.dispatch task 7818af2b-1172-4a4f-8654-4af8494abb6a is finished
2021-05-03 23:35:54,490 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(e76e1823-4a6b-4369-bf4b-5ce57f392479, queue=dev-awx-0)
[2021-05-03 23:35:54,490: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(e76e1823-4a6b-4369-bf4b-5ce57f392479, queue=dev-awx-0)
2021-05-03 23:35:54,513 DEBUG awx.main.dispatch delivered e76e1823-4a6b-4369-bf4b-5ce57f392479 to worker[202] qsize 0
2021-05-03 23:35:54,518 DEBUG awx.main.dispatch task e76e1823-4a6b-4369-bf4b-5ce57f392479 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-05-03 23:35:54,521 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-05-03 23:35:54,543 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 470, in awx_isolated_heartbeat
for isolated_instance in isolated_instance_qs:
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-03 23:35:54,545 DEBUG awx.main.dispatch task e76e1823-4a6b-4369-bf4b-5ce57f392479 is finished
RESULT 2
OKREADY
2021-05-03 23:36:07,427 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8680e87b-1d3d-4f1a-8b78-522d7f01f46c, queue=awx_private_queue)
[2021-05-03 23:36:07,427: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8680e87b-1d3d-4f1a-8b78-522d7f01f46c, queue=awx_private_queue)
2021-05-03 23:36:07,454 DEBUG awx.main.dispatch delivered 8680e87b-1d3d-4f1a-8b78-522d7f01f46c to worker[197] qsize 0
2021-05-03 23:36:07,467 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d74e7665-4638-4c22-9914-ff34fbf5d7f3, queue=dev-awx-0)
[2021-05-03 23:36:07,467: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d74e7665-4638-4c22-9914-ff34fbf5d7f3, queue=dev-awx-0)
2021-05-03 23:36:07,462 DEBUG awx.main.dispatch task 8680e87b-1d3d-4f1a-8b78-522d7f01f46c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:36:07,484 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(47ed5163-4d37-4bf4-a807-29df01ce3428, queue=dev-awx-0)
[2021-05-03 23:36:07,484: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(47ed5163-4d37-4bf4-a807-29df01ce3428, queue=dev-awx-0)
2021-05-03 23:36:07,595 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fe8c33d0-82a8-4e9e-a013-121e8a4d96e3, queue=awx_private_queue)
[2021-05-03 23:36:07,595: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fe8c33d0-82a8-4e9e-a013-121e8a4d96e3, queue=awx_private_queue)
2021-05-03 23:36:07,601 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:36:07,660 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:36:07,664 DEBUG awx.main.dispatch task 8680e87b-1d3d-4f1a-8b78-522d7f01f46c is finished
2021-05-03 23:36:25,154 DEBUG awx.main.dispatch delivered d74e7665-4638-4c22-9914-ff34fbf5d7f3 to worker[202] qsize 0
2021-05-03 23:36:25,162 DEBUG awx.main.dispatch delivered 47ed5163-4d37-4bf4-a807-29df01ce3428 to worker[208] qsize 0
2021-05-03 23:36:25,165 DEBUG awx.main.dispatch delivered fe8c33d0-82a8-4e9e-a013-121e8a4d96e3 to worker[197] qsize 0
2021-05-03 23:36:25,173 DEBUG awx.main.dispatch task d74e7665-4638-4c22-9914-ff34fbf5d7f3 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-03 23:36:25,189 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-03 23:36:25,213 DEBUG awx.main.dispatch task fe8c33d0-82a8-4e9e-a013-121e8a4d96e3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:36:25,218 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:36:25,241 DEBUG awx.main.dispatch task 47ed5163-4d37-4bf4-a807-29df01ce3428 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-03 23:36:25,270 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-03 23:36:25,275 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140202230842944_6" does not exist
2021-05-03 23:36:25,278 DEBUG awx.main.dispatch task 47ed5163-4d37-4bf4-a807-29df01ce3428 is finished
2021-05-03 23:36:25,274 DEBUG awx.main.dispatch task d74e7665-4638-4c22-9914-ff34fbf5d7f3 is finished
2021-05-03 23:36:25,328 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:36:25,332 DEBUG awx.main.dispatch task fe8c33d0-82a8-4e9e-a013-121e8a4d96e3 is finished
2021-05-03 23:36:27,612 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8201ad31-1d15-494e-ac5b-423cd9368765, queue=awx_private_queue)
[2021-05-03 23:36:27,612: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8201ad31-1d15-494e-ac5b-423cd9368765, queue=awx_private_queue)
2021-05-03 23:36:27,640 DEBUG awx.main.dispatch delivered 8201ad31-1d15-494e-ac5b-423cd9368765 to worker[208] qsize 0
2021-05-03 23:36:27,650 DEBUG awx.main.dispatch task 8201ad31-1d15-494e-ac5b-423cd9368765 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:36:27,651 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:36:27,671 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:36:27,673 DEBUG awx.main.dispatch task 8201ad31-1d15-494e-ac5b-423cd9368765 is finished
2021-05-03 23:36:37,426 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(fe862257-dc23-4157-b1a8-2ea8f3fa8e64, queue=awx_private_queue)
[2021-05-03 23:36:37,426: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(fe862257-dc23-4157-b1a8-2ea8f3fa8e64, queue=awx_private_queue)
2021-05-03 23:36:37,444 DEBUG awx.main.dispatch delivered fe862257-dc23-4157-b1a8-2ea8f3fa8e64 to worker[202] qsize 0
2021-05-03 23:36:37,451 DEBUG awx.main.dispatch task fe862257-dc23-4157-b1a8-2ea8f3fa8e64 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-03 23:36:37,498 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-03 23:36:37,503 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-03 23:36:37,505 DEBUG awx.main.dispatch task fe862257-dc23-4157-b1a8-2ea8f3fa8e64 is finished
2021-05-03 23:36:47,606 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ed6ef2cf-c45f-4b8f-a40d-dab8b8cf0288, queue=awx_private_queue)
[2021-05-03 23:36:47,606: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ed6ef2cf-c45f-4b8f-a40d-dab8b8cf0288, queue=awx_private_queue)
2021-05-03 23:36:47,631 DEBUG awx.main.dispatch delivered ed6ef2cf-c45f-4b8f-a40d-dab8b8cf0288 to worker[204] qsize 0
2021-05-03 23:36:47,639 DEBUG awx.main.dispatch task ed6ef2cf-c45f-4b8f-a40d-dab8b8cf0288 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-03 23:36:47,646 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-03 23:36:47,671 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-03 23:36:47,674 DEBUG awx.main.dispatch task ed6ef2cf-c45f-4b8f-a40d-dab8b8cf0288 is finished
RESULT 2
OKREADY
2021-05-03 23:37:07,440 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(cefcac18-5054-4f90-9c6c-0cca9cc273fe, queue=awx_private_queue)
[2021-05-03 23:37:07,440: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(cefcac18-5054-4f90-9c6c-0cca9cc273fe, queue=awx_private_queue)
2021-05-03 23:37:07,512 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(ffc8c0c6-bac5-419a-8a7c-52d16c6e71f5, queue=dev-awx-0)
[2021-05-03 23:37:07,512: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(ffc8c0c6-bac5-419a-8a7c-52d16c6e71f5, queue=dev-awx-0)
2021-05-03 23:37:07,555 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3e62482d-db19-40d7-9f32-ed9214765080, queue=dev-awx-0)
[2021-05-03 23:37:07,555: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3e62482d-db19-40d7-9f32-ed9214765080, queue=dev-awx-0)
2021-05-03 23:37:07,608 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(536945cf-4a26-43ed-96d2-6bf00028b1a5, queue=awx_private_queue)
[2021-05-03 23:37:07,608: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(536945cf-4a26-43ed-96d2-6bf00028b1a5, queue=awx_private_queue)
2021-05-03 23:37:07,509 DEBUG awx.main.dispatch delivered cefcac18-5054-4f90-9c6c-0cca9cc273fe to worker[208] qsize 0
2021-05-03 23:37:07,765 DEBUG awx.main.dispatch t