Using /etc/ansible/ansible.cfg as config file 127.0.0.1 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/libexec/platform-python" }, "changed": false, "elapsed": 0, "match_groupdict": {}, "match_groups": [], "path": null, "port": 5432, "search_regex": null, "state": "started" } Using /etc/ansible/ansible.cfg as config file 127.0.0.1 | FAILED! => { "ansible_facts": { "discovered_interpreter_python": "/usr/libexec/platform-python" }, "changed": false, "msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int" } Using /etc/ansible/ansible.cfg as config file 127.0.0.1 | FAILED! => { "ansible_facts": { "discovered_interpreter_python": "/usr/libexec/platform-python" }, "changed": false, "msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int" } Using /etc/ansible/ansible.cfg as config file 127.0.0.1 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/libexec/platform-python" }, "changed": false, "db": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER" } Default organization added. Demo Credential, Inventory, and Job Template added. (changed: True) 2021-04-08 14:37:24,734 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(036690e5-1838-4f0c-b29d-57e6688997dd, queue=awx_private_queue) Successfully registered instance dev-awx-0 (changed: True) 2021-04-08 14:37:30,266 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(1a2d1dcb-e69f-4962-938b-c37c326b011b, queue=awx_private_queue) 2021-04-08 14:37:30,361 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(09d7bf9d-d753-464e-bf32-a2a830f87a8e, queue=awx_private_queue) Creating instance group tower (changed: True) 2021-04-08 14:37:31,264 INFO RPC interface 'supervisor' initialized 2021-04-08 14:37:31,265 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2021-04-08 14:37:31,265 INFO supervisord started with pid 150 2021-04-08 14:37:32,269 INFO spawned: 'awx-config-watcher' with pid 153 2021-04-08 14:37:32,274 INFO spawned: 'dispatcher' with pid 154 2021-04-08 14:37:32,278 INFO spawned: 'callback-receiver' with pid 155 2021-04-08 14:37:32,281 INFO spawned: 'channels-worker' with pid 156 READY 2021-04-08 14:37:33,421 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-04-08 14:37:33,421 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-04-08 14:37:33,421 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-04-08 14:37:33,421 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-04-08 14:37:35,703 ERROR celery.beat Removing corrupted schedule file '/var/lib/awx/beat.db': error(11, 'Resource temporarily unavailable') Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 485, in setup_schedule self._store = self._open_schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 475, in _open_schedule return self.persistence.open(self.schedule_filename, writeback=True) File "/usr/lib64/python3.6/shelve.py", line 243, in open return DbfilenameShelf(filename, flag, protocol, writeback) File "/usr/lib64/python3.6/shelve.py", line 227, in __init__ Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback) File "/usr/lib64/python3.6/dbm/__init__.py", line 94, in open return mod.open(file, flag, mode) _gdbm.error: [Errno 11] Resource temporarily unavailable 2021-04-08 14:37:35,718 WARNING awx.main.dispatch scaling up worker pid:176 2021-04-08 14:37:35,729 WARNING awx.main.dispatch scaling up worker pid:177 2021-04-08 14:37:35,741 WARNING awx.main.dispatch scaling up worker pid:178 2021-04-08 14:37:35,753 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179 2021-04-08 14:37:35,753 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179 2021-04-08 14:37:35,762 WARNING awx.main.dispatch scaling up worker pid:180 2021-04-08 14:37:35,771 DEBUG awx.main.tasks Syncing Schedules 2021-04-08 14:37:35,775 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:181 2021-04-08 14:37:35,775 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:181 2021-04-08 14:37:35,788 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:182 2021-04-08 14:37:35,788 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:182 2021-04-08 14:37:35,801 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:183 2021-04-08 14:37:35,801 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:183 2021-04-08 14:37:35,815 - INFO - runworker - Using single-threaded worker. 2021-04-08 14:37:35,815 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer) 2021-04-08 14:37:35,817 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive 2021-04-08 14:37:40,147 DEBUG awx.main.tasks Waited 0.001232147216796875 seconds to obtain lock name: cluster_policy_lock 2021-04-08 14:37:40,168 DEBUG awx.main.tasks Total non-isolated instances:1 available for policy: 1 2021-04-08 14:37:40,168 DEBUG awx.main.tasks Policy percentage, adding Instances [1] to Group tower 2021-04-08 14:37:40,168 DEBUG awx.main.tasks Adding instances [1] to group tower 2021-04-08 14:37:41,850 DEBUG awx.main.tasks Cluster policy computation finished in 1.702770471572876 seconds 2021-04-08 14:37:41,851 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:37:43,331 DEBUG awx.main.dispatch listening on ['awx_private_queue [direct]', 'dev-awx-0 [direct]', 'tower_broadcast_all_dev-awx-0 [fanout]'] 2021-04-08 14:37:55,886 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(eaebdefe-90e6-4cbf-8e06-9ea3f50283df, queue=awx_private_queue) [2021-04-08 14:37:55,886: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(eaebdefe-90e6-4cbf-8e06-9ea3f50283df, queue=awx_private_queue) 2021-04-08 14:37:55,903 DEBUG awx.main.dispatch delivered eaebdefe-90e6-4cbf-8e06-9ea3f50283df to worker[177] qsize 0 2021-04-08 14:37:55,907 DEBUG awx.main.dispatch task eaebdefe-90e6-4cbf-8e06-9ea3f50283df starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:37:55,910 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:37:55,961 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:37:56,062 DEBUG awx.main.dispatch task eaebdefe-90e6-4cbf-8e06-9ea3f50283df is finished RESULT 2 OKREADY 2021-04-08 14:38:05,866 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(23b2e475-2644-41d9-a91c-0c5fccb05e79, queue=awx_private_queue) [2021-04-08 14:38:05,866: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(23b2e475-2644-41d9-a91c-0c5fccb05e79, queue=awx_private_queue) 2021-04-08 14:38:05,887 DEBUG awx.main.dispatch delivered 23b2e475-2644-41d9-a91c-0c5fccb05e79 to worker[176] qsize 0 2021-04-08 14:38:05,895 DEBUG awx.main.dispatch task 23b2e475-2644-41d9-a91c-0c5fccb05e79 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:38:05,934 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:38:06,053 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:38:05.944304+00:00 2021-04-08 14:38:06,078 DEBUG awx.main.dispatch task 23b2e475-2644-41d9-a91c-0c5fccb05e79 is finished 2021-04-08 14:38:15,874 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3e9f68d7-b07b-4b74-8cf3-cd4cbcea7b84, queue=awx_private_queue) [2021-04-08 14:38:15,874: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3e9f68d7-b07b-4b74-8cf3-cd4cbcea7b84, queue=awx_private_queue) 2021-04-08 14:38:15,907 DEBUG awx.main.dispatch delivered 3e9f68d7-b07b-4b74-8cf3-cd4cbcea7b84 to worker[178] qsize 0 2021-04-08 14:38:15,916 DEBUG awx.main.dispatch task 3e9f68d7-b07b-4b74-8cf3-cd4cbcea7b84 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:38:15,920 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:38:15,949 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:38:16,017 DEBUG awx.main.dispatch task 3e9f68d7-b07b-4b74-8cf3-cd4cbcea7b84 is finished 2021-04-08 14:38:35,871 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(877389c4-80bd-4c3f-9878-7f5f1669e0ab, queue=dev-awx-0) [2021-04-08 14:38:35,871: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(877389c4-80bd-4c3f-9878-7f5f1669e0ab, queue=dev-awx-0) 2021-04-08 14:38:36,089 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b06512a1-52a0-44a4-9d48-84bac69e21d0, queue=dev-awx-0) [2021-04-08 14:38:36,089: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b06512a1-52a0-44a4-9d48-84bac69e21d0, queue=dev-awx-0) 2021-04-08 14:38:36,106 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(47628018-b502-4409-954c-97cd7c12987c, queue=awx_private_queue) [2021-04-08 14:38:36,106: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(47628018-b502-4409-954c-97cd7c12987c, queue=awx_private_queue) 2021-04-08 14:38:36,108 DEBUG awx.main.dispatch delivered 877389c4-80bd-4c3f-9878-7f5f1669e0ab to worker[176] qsize 0 2021-04-08 14:38:36,111 DEBUG awx.main.dispatch delivered b06512a1-52a0-44a4-9d48-84bac69e21d0 to worker[178] qsize 0 2021-04-08 14:38:36,111 DEBUG awx.main.dispatch task 877389c4-80bd-4c3f-9878-7f5f1669e0ab starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:38:36,114 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:38:36,113 DEBUG awx.main.dispatch task b06512a1-52a0-44a4-9d48-84bac69e21d0 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:38:36,124 DEBUG awx.main.dispatch delivered 47628018-b502-4409-954c-97cd7c12987c to worker[180] qsize 0 2021-04-08 14:38:36,125 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(57b80d52-05c2-476f-a2d8-e8b99d36e05e, queue=awx_private_queue) [2021-04-08 14:38:36,125: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(57b80d52-05c2-476f-a2d8-e8b99d36e05e, queue=awx_private_queue) 2021-04-08 14:38:36,130 DEBUG awx.main.dispatch task b06512a1-52a0-44a4-9d48-84bac69e21d0 is finished 2021-04-08 14:38:36,129 DEBUG awx.main.dispatch task 47628018-b502-4409-954c-97cd7c12987c starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:38:36,140 DEBUG awx.main.dispatch delivered 57b80d52-05c2-476f-a2d8-e8b99d36e05e to worker[177] qsize 0 2021-04-08 14:38:36,144 DEBUG awx.main.dispatch task 57b80d52-05c2-476f-a2d8-e8b99d36e05e starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:38:36,150 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:38:36,150 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:38:36,156 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:38:05.934854+00:00 2021-04-08 14:38:36,173 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:38:36,225 DEBUG awx.main.dispatch task 57b80d52-05c2-476f-a2d8-e8b99d36e05e is finished 2021-04-08 14:38:36,779 DEBUG awx.main.dispatch task 877389c4-80bd-4c3f-9878-7f5f1669e0ab is finished 2021-04-08 14:38:37,224 DEBUG awx.main.dispatch task 47628018-b502-4409-954c-97cd7c12987c is finished 2021-04-08 14:38:56,134 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c9a5d2ab-9900-45e6-a3e4-28079e148f7e, queue=awx_private_queue) [2021-04-08 14:38:56,134: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c9a5d2ab-9900-45e6-a3e4-28079e148f7e, queue=awx_private_queue) 2021-04-08 14:38:56,158 DEBUG awx.main.dispatch delivered c9a5d2ab-9900-45e6-a3e4-28079e148f7e to worker[178] qsize 0 2021-04-08 14:38:56,161 DEBUG awx.main.dispatch task c9a5d2ab-9900-45e6-a3e4-28079e148f7e starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:38:56,162 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:38:56,205 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:38:56,238 DEBUG awx.main.dispatch task c9a5d2ab-9900-45e6-a3e4-28079e148f7e is finished RESULT 2 OKREADY 2021-04-08 14:39:06,108 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6a72fbbd-88d2-446c-bf9b-3d3cdf7310f7, queue=awx_private_queue) [2021-04-08 14:39:06,108: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6a72fbbd-88d2-446c-bf9b-3d3cdf7310f7, queue=awx_private_queue) 2021-04-08 14:39:06,125 DEBUG awx.main.dispatch delivered 6a72fbbd-88d2-446c-bf9b-3d3cdf7310f7 to worker[176] qsize 0 2021-04-08 14:39:06,128 DEBUG awx.main.dispatch task 6a72fbbd-88d2-446c-bf9b-3d3cdf7310f7 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:39:06,149 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:39:06,152 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:38:36.150661+00:00 2021-04-08 14:39:06,855 DEBUG awx.main.dispatch task 6a72fbbd-88d2-446c-bf9b-3d3cdf7310f7 is finished 2021-04-08 14:39:16,134 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e0270a8a-3f1a-4207-ab62-23fbff8b4915, queue=awx_private_queue) [2021-04-08 14:39:16,134: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e0270a8a-3f1a-4207-ab62-23fbff8b4915, queue=awx_private_queue) 2021-04-08 14:39:16,151 DEBUG awx.main.dispatch delivered e0270a8a-3f1a-4207-ab62-23fbff8b4915 to worker[178] qsize 0 2021-04-08 14:39:16,154 DEBUG awx.main.dispatch task e0270a8a-3f1a-4207-ab62-23fbff8b4915 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:39:16,156 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:39:16,181 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:39:16,224 DEBUG awx.main.dispatch task e0270a8a-3f1a-4207-ab62-23fbff8b4915 is finished 2021-04-08 14:39:35,875 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5ddb65bd-a7e4-4e27-9202-1c40ce84eb6d, queue=dev-awx-0) [2021-04-08 14:39:35,875: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5ddb65bd-a7e4-4e27-9202-1c40ce84eb6d, queue=dev-awx-0) 2021-04-08 14:39:36,107 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(9868b7a0-4e95-4cb8-a3a6-93dac5d08645, queue=dev-awx-0) [2021-04-08 14:39:36,107: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(9868b7a0-4e95-4cb8-a3a6-93dac5d08645, queue=dev-awx-0) 2021-04-08 14:39:36,127 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(17817407-972e-4370-a0be-5036c6f717e3, queue=awx_private_queue) [2021-04-08 14:39:36,127: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(17817407-972e-4370-a0be-5036c6f717e3, queue=awx_private_queue) 2021-04-08 14:39:36,129 DEBUG awx.main.dispatch delivered 5ddb65bd-a7e4-4e27-9202-1c40ce84eb6d to worker[180] qsize 0 2021-04-08 14:39:36,132 DEBUG awx.main.dispatch delivered 9868b7a0-4e95-4cb8-a3a6-93dac5d08645 to worker[177] qsize 0 2021-04-08 14:39:36,132 DEBUG awx.main.dispatch task 5ddb65bd-a7e4-4e27-9202-1c40ce84eb6d starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:39:36,136 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:39:36,136 DEBUG awx.main.dispatch task 9868b7a0-4e95-4cb8-a3a6-93dac5d08645 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:39:36,154 DEBUG awx.main.dispatch delivered 17817407-972e-4370-a0be-5036c6f717e3 to worker[176] qsize 0 2021-04-08 14:39:36,155 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(14d7492d-15dd-4eec-af01-ccf5b8cb80a4, queue=awx_private_queue) [2021-04-08 14:39:36,155: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(14d7492d-15dd-4eec-af01-ccf5b8cb80a4, queue=awx_private_queue) 2021-04-08 14:39:36,157 DEBUG awx.main.dispatch task 17817407-972e-4370-a0be-5036c6f717e3 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:39:36,169 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:39:36,172 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:39:06.149437+00:00 2021-04-08 14:39:36,176 DEBUG awx.main.dispatch delivered 14d7492d-15dd-4eec-af01-ccf5b8cb80a4 to worker[178] qsize 0 2021-04-08 14:39:36,179 DEBUG awx.main.dispatch task 14d7492d-15dd-4eec-af01-ccf5b8cb80a4 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:39:36,183 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:39:36,200 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:39:36,206 DEBUG awx.main.dispatch task 9868b7a0-4e95-4cb8-a3a6-93dac5d08645 is finished 2021-04-08 14:39:36,227 DEBUG awx.main.dispatch task 14d7492d-15dd-4eec-af01-ccf5b8cb80a4 is finished 2021-04-08 14:39:36,902 DEBUG awx.main.dispatch task 5ddb65bd-a7e4-4e27-9202-1c40ce84eb6d is finished 2021-04-08 14:39:37,890 DEBUG awx.main.dispatch task 17817407-972e-4370-a0be-5036c6f717e3 is finished 2021-04-08 14:39:56,172 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(93701ca2-7184-46e8-927a-0c97f2a7ddfd, queue=awx_private_queue) [2021-04-08 14:39:56,172: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(93701ca2-7184-46e8-927a-0c97f2a7ddfd, queue=awx_private_queue) 2021-04-08 14:39:56,207 DEBUG awx.main.dispatch delivered 93701ca2-7184-46e8-927a-0c97f2a7ddfd to worker[180] qsize 0 2021-04-08 14:39:56,220 DEBUG awx.main.dispatch task 93701ca2-7184-46e8-927a-0c97f2a7ddfd starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:39:56,222 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:39:56,246 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:39:56,294 DEBUG awx.main.dispatch task 93701ca2-7184-46e8-927a-0c97f2a7ddfd is finished RESULT 2 OKREADY 2021-04-08 14:40:06,127 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e6891906-e5f3-44c3-b2aa-3566754918be, queue=awx_private_queue) [2021-04-08 14:40:06,127: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e6891906-e5f3-44c3-b2aa-3566754918be, queue=awx_private_queue) 2021-04-08 14:40:06,142 DEBUG awx.main.dispatch delivered e6891906-e5f3-44c3-b2aa-3566754918be to worker[180] qsize 0 2021-04-08 14:40:06,144 DEBUG awx.main.dispatch task e6891906-e5f3-44c3-b2aa-3566754918be starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:40:06,153 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:40:06,156 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:39:36.169328+00:00 2021-04-08 14:40:06,915 DEBUG awx.main.dispatch task e6891906-e5f3-44c3-b2aa-3566754918be is finished 2021-04-08 14:40:16,170 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0b0ea9d3-c3c3-45fe-befc-49fe5a474e65, queue=awx_private_queue) [2021-04-08 14:40:16,170: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0b0ea9d3-c3c3-45fe-befc-49fe5a474e65, queue=awx_private_queue) 2021-04-08 14:40:16,203 DEBUG awx.main.dispatch delivered 0b0ea9d3-c3c3-45fe-befc-49fe5a474e65 to worker[180] qsize 0 2021-04-08 14:40:16,206 DEBUG awx.main.dispatch task 0b0ea9d3-c3c3-45fe-befc-49fe5a474e65 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:40:16,207 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:40:16,222 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:40:16,251 DEBUG awx.main.dispatch task 0b0ea9d3-c3c3-45fe-befc-49fe5a474e65 is finished 2021-04-08 14:40:35,875 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(32f69f3e-b6ed-40bd-8506-d5f9c5665c67, queue=dev-awx-0) [2021-04-08 14:40:35,875: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(32f69f3e-b6ed-40bd-8506-d5f9c5665c67, queue=dev-awx-0) 2021-04-08 14:40:35,917 DEBUG awx.main.dispatch delivered 32f69f3e-b6ed-40bd-8506-d5f9c5665c67 to worker[178] qsize 0 2021-04-08 14:40:35,920 DEBUG awx.main.dispatch task 32f69f3e-b6ed-40bd-8506-d5f9c5665c67 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:40:35,942 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:40:36,106 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2d62b973-0020-4d37-a4e9-3e37690fbbcc, queue=dev-awx-0) [2021-04-08 14:40:36,106: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2d62b973-0020-4d37-a4e9-3e37690fbbcc, queue=dev-awx-0) 2021-04-08 14:40:36,119 DEBUG awx.main.dispatch delivered 2d62b973-0020-4d37-a4e9-3e37690fbbcc to worker[177] qsize 0 2021-04-08 14:40:36,122 DEBUG awx.main.dispatch task 2d62b973-0020-4d37-a4e9-3e37690fbbcc starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:40:36,127 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0319b3c5-6bde-4d1b-b7b6-7a039f62cc21, queue=awx_private_queue) [2021-04-08 14:40:36,127: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0319b3c5-6bde-4d1b-b7b6-7a039f62cc21, queue=awx_private_queue) 2021-04-08 14:40:36,139 DEBUG awx.main.dispatch task 2d62b973-0020-4d37-a4e9-3e37690fbbcc is finished 2021-04-08 14:40:36,168 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7171cd40-4a89-407d-be2e-fbdba031eba7, queue=awx_private_queue) [2021-04-08 14:40:36,168: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7171cd40-4a89-407d-be2e-fbdba031eba7, queue=awx_private_queue) 2021-04-08 14:40:36,140 DEBUG awx.main.dispatch delivered 0319b3c5-6bde-4d1b-b7b6-7a039f62cc21 to worker[180] qsize 0 2021-04-08 14:40:36,314 DEBUG awx.main.dispatch task 0319b3c5-6bde-4d1b-b7b6-7a039f62cc21 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:40:36,315 DEBUG awx.main.dispatch delivered 7171cd40-4a89-407d-be2e-fbdba031eba7 to worker[176] qsize 0 2021-04-08 14:40:36,319 DEBUG awx.main.dispatch task 7171cd40-4a89-407d-be2e-fbdba031eba7 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:40:36,322 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:40:36,325 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:40:36,328 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:40:06.153642+00:00 2021-04-08 14:40:36,342 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:40:36,387 DEBUG awx.main.dispatch task 7171cd40-4a89-407d-be2e-fbdba031eba7 is finished 2021-04-08 14:40:36,432 DEBUG awx.main.dispatch task 32f69f3e-b6ed-40bd-8506-d5f9c5665c67 is finished 2021-04-08 14:40:37,731 DEBUG awx.main.dispatch task 0319b3c5-6bde-4d1b-b7b6-7a039f62cc21 is finished 2021-04-08 14:40:56,179 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(51bb71c5-03dc-4cf3-b966-2028e36e467b, queue=awx_private_queue) [2021-04-08 14:40:56,179: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(51bb71c5-03dc-4cf3-b966-2028e36e467b, queue=awx_private_queue) 2021-04-08 14:40:56,193 DEBUG awx.main.dispatch delivered 51bb71c5-03dc-4cf3-b966-2028e36e467b to worker[177] qsize 0 2021-04-08 14:40:56,195 DEBUG awx.main.dispatch task 51bb71c5-03dc-4cf3-b966-2028e36e467b starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:40:56,197 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:40:56,215 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:40:56,244 DEBUG awx.main.dispatch task 51bb71c5-03dc-4cf3-b966-2028e36e467b is finished RESULT 2 OKREADY 2021-04-08 14:41:06,128 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1caf46b7-fd68-4f66-92d4-ef7bf4245dae, queue=awx_private_queue) [2021-04-08 14:41:06,128: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1caf46b7-fd68-4f66-92d4-ef7bf4245dae, queue=awx_private_queue) 2021-04-08 14:41:06,144 DEBUG awx.main.dispatch delivered 1caf46b7-fd68-4f66-92d4-ef7bf4245dae to worker[178] qsize 0 2021-04-08 14:41:06,146 DEBUG awx.main.dispatch task 1caf46b7-fd68-4f66-92d4-ef7bf4245dae starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:41:06,156 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:41:06,160 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:40:36.325860+00:00 2021-04-08 14:41:07,695 DEBUG awx.main.dispatch task 1caf46b7-fd68-4f66-92d4-ef7bf4245dae is finished 2021-04-08 14:41:16,179 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a9e722d4-1d3f-4b02-9b8d-290b093b5a13, queue=awx_private_queue) [2021-04-08 14:41:16,179: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a9e722d4-1d3f-4b02-9b8d-290b093b5a13, queue=awx_private_queue) 2021-04-08 14:41:16,193 DEBUG awx.main.dispatch delivered a9e722d4-1d3f-4b02-9b8d-290b093b5a13 to worker[177] qsize 0 2021-04-08 14:41:16,195 DEBUG awx.main.dispatch task a9e722d4-1d3f-4b02-9b8d-290b093b5a13 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:41:16,196 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:41:16,218 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:41:16,247 DEBUG awx.main.dispatch task a9e722d4-1d3f-4b02-9b8d-290b093b5a13 is finished 2021-04-08 14:41:35,881 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(04636d02-a945-41d5-bcf2-a77e727edc9e, queue=dev-awx-0) [2021-04-08 14:41:35,881: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(04636d02-a945-41d5-bcf2-a77e727edc9e, queue=dev-awx-0) 2021-04-08 14:41:35,908 DEBUG awx.main.dispatch delivered 04636d02-a945-41d5-bcf2-a77e727edc9e to worker[176] qsize 0 2021-04-08 14:41:35,910 DEBUG awx.main.dispatch task 04636d02-a945-41d5-bcf2-a77e727edc9e starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:41:35,911 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:41:36,106 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(fa62a0f3-fa46-44ef-900a-f01c1cc3c3b9, queue=dev-awx-0) [2021-04-08 14:41:36,106: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(fa62a0f3-fa46-44ef-900a-f01c1cc3c3b9, queue=dev-awx-0) 2021-04-08 14:41:36,136 DEBUG awx.main.dispatch delivered fa62a0f3-fa46-44ef-900a-f01c1cc3c3b9 to worker[177] qsize 0 2021-04-08 14:41:36,137 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(398f0a07-1bcc-4b09-8425-e4d7d0bc3ba8, queue=awx_private_queue) [2021-04-08 14:41:36,137: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(398f0a07-1bcc-4b09-8425-e4d7d0bc3ba8, queue=awx_private_queue) 2021-04-08 14:41:36,138 DEBUG awx.main.dispatch task fa62a0f3-fa46-44ef-900a-f01c1cc3c3b9 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:41:36,150 DEBUG awx.main.dispatch task fa62a0f3-fa46-44ef-900a-f01c1cc3c3b9 is finished 2021-04-08 14:41:36,179 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9b57a8dd-7dad-4b6d-a866-724897476f61, queue=awx_private_queue) [2021-04-08 14:41:36,179: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9b57a8dd-7dad-4b6d-a866-724897476f61, queue=awx_private_queue) 2021-04-08 14:41:36,155 DEBUG awx.main.dispatch delivered 398f0a07-1bcc-4b09-8425-e4d7d0bc3ba8 to worker[177] qsize 0 2021-04-08 14:41:36,328 DEBUG awx.main.dispatch delivered 9b57a8dd-7dad-4b6d-a866-724897476f61 to worker[180] qsize 0 2021-04-08 14:41:36,328 DEBUG awx.main.dispatch task 398f0a07-1bcc-4b09-8425-e4d7d0bc3ba8 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:41:36,329 DEBUG awx.main.dispatch task 9b57a8dd-7dad-4b6d-a866-724897476f61 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:41:36,331 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:41:36,337 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:41:36,340 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:41:06.157105+00:00 2021-04-08 14:41:36,345 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:41:36,376 DEBUG awx.main.dispatch task 9b57a8dd-7dad-4b6d-a866-724897476f61 is finished 2021-04-08 14:41:36,992 DEBUG awx.main.dispatch task 04636d02-a945-41d5-bcf2-a77e727edc9e is finished 2021-04-08 14:41:37,769 DEBUG awx.main.dispatch task 398f0a07-1bcc-4b09-8425-e4d7d0bc3ba8 is finished 2021-04-08 14:41:56,195 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6e269f61-3615-4700-988e-eca58735c779, queue=awx_private_queue) [2021-04-08 14:41:56,195: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6e269f61-3615-4700-988e-eca58735c779, queue=awx_private_queue) 2021-04-08 14:41:56,223 DEBUG awx.main.dispatch delivered 6e269f61-3615-4700-988e-eca58735c779 to worker[180] qsize 0 2021-04-08 14:41:56,231 DEBUG awx.main.dispatch task 6e269f61-3615-4700-988e-eca58735c779 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:41:56,234 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:41:56,257 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:41:56,298 DEBUG awx.main.dispatch task 6e269f61-3615-4700-988e-eca58735c779 is finished RESULT 2 OKREADY 2021-04-08 14:42:06,138 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(def3efd1-611f-4fee-b68e-914bcc2eeb95, queue=awx_private_queue) [2021-04-08 14:42:06,138: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(def3efd1-611f-4fee-b68e-914bcc2eeb95, queue=awx_private_queue) 2021-04-08 14:42:06,156 DEBUG awx.main.dispatch delivered def3efd1-611f-4fee-b68e-914bcc2eeb95 to worker[177] qsize 0 2021-04-08 14:42:06,161 DEBUG awx.main.dispatch task def3efd1-611f-4fee-b68e-914bcc2eeb95 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:42:06,177 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:42:06,182 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:41:36.337789+00:00 2021-04-08 14:42:06,277 DEBUG awx.main.dispatch task def3efd1-611f-4fee-b68e-914bcc2eeb95 is finished 2021-04-08 14:42:16,191 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d542e919-55ff-4a1b-a731-be419344c550, queue=awx_private_queue) [2021-04-08 14:42:16,191: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d542e919-55ff-4a1b-a731-be419344c550, queue=awx_private_queue) 2021-04-08 14:42:16,208 DEBUG awx.main.dispatch delivered d542e919-55ff-4a1b-a731-be419344c550 to worker[176] qsize 0 2021-04-08 14:42:16,211 DEBUG awx.main.dispatch task d542e919-55ff-4a1b-a731-be419344c550 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:42:16,215 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:42:16,240 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:42:16,273 DEBUG awx.main.dispatch task d542e919-55ff-4a1b-a731-be419344c550 is finished 2021-04-08 14:42:35,883 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6238bdaf-3754-484c-ac34-3ed9e93f9b75, queue=dev-awx-0) [2021-04-08 14:42:35,883: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6238bdaf-3754-484c-ac34-3ed9e93f9b75, queue=dev-awx-0) 2021-04-08 14:42:35,921 DEBUG awx.main.dispatch delivered 6238bdaf-3754-484c-ac34-3ed9e93f9b75 to worker[176] qsize 0 2021-04-08 14:42:35,923 DEBUG awx.main.dispatch task 6238bdaf-3754-484c-ac34-3ed9e93f9b75 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:42:35,925 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:42:36,106 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(327636db-ae2a-4926-8f6b-986ac7615de9, queue=dev-awx-0) [2021-04-08 14:42:36,106: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(327636db-ae2a-4926-8f6b-986ac7615de9, queue=dev-awx-0) 2021-04-08 14:42:36,129 DEBUG awx.main.dispatch task 6238bdaf-3754-484c-ac34-3ed9e93f9b75 is finished 2021-04-08 14:42:36,141 DEBUG awx.main.dispatch delivered 327636db-ae2a-4926-8f6b-986ac7615de9 to worker[177] qsize 0 2021-04-08 14:42:36,142 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(306a569c-46bc-4840-97db-dfc46a851b68, queue=awx_private_queue) [2021-04-08 14:42:36,142: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(306a569c-46bc-4840-97db-dfc46a851b68, queue=awx_private_queue) 2021-04-08 14:42:36,143 DEBUG awx.main.dispatch task 327636db-ae2a-4926-8f6b-986ac7615de9 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:42:36,157 DEBUG awx.main.dispatch task 327636db-ae2a-4926-8f6b-986ac7615de9 is finished 2021-04-08 14:42:36,192 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(002aeba3-cbb4-4f13-af33-7d04004aa115, queue=awx_private_queue) [2021-04-08 14:42:36,192: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(002aeba3-cbb4-4f13-af33-7d04004aa115, queue=awx_private_queue) 2021-04-08 14:42:36,158 DEBUG awx.main.dispatch delivered 306a569c-46bc-4840-97db-dfc46a851b68 to worker[178] qsize 0 2021-04-08 14:42:36,352 DEBUG awx.main.dispatch delivered 002aeba3-cbb4-4f13-af33-7d04004aa115 to worker[180] qsize 0 2021-04-08 14:42:36,351 DEBUG awx.main.dispatch task 306a569c-46bc-4840-97db-dfc46a851b68 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:42:36,355 DEBUG awx.main.dispatch task 002aeba3-cbb4-4f13-af33-7d04004aa115 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:42:36,358 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:42:36,367 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:42:36,371 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:42:06.177537+00:00 2021-04-08 14:42:36,377 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:42:36,405 DEBUG awx.main.dispatch task 002aeba3-cbb4-4f13-af33-7d04004aa115 is finished 2021-04-08 14:42:36,438 DEBUG awx.main.dispatch task 306a569c-46bc-4840-97db-dfc46a851b68 is finished 2021-04-08 14:42:56,202 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fe70848e-0573-4e64-829e-15726fdf255d, queue=awx_private_queue) [2021-04-08 14:42:56,202: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fe70848e-0573-4e64-829e-15726fdf255d, queue=awx_private_queue) 2021-04-08 14:42:56,217 DEBUG awx.main.dispatch delivered fe70848e-0573-4e64-829e-15726fdf255d to worker[176] qsize 0 2021-04-08 14:42:56,219 DEBUG awx.main.dispatch task fe70848e-0573-4e64-829e-15726fdf255d starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:42:56,221 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:42:56,239 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:42:56,275 DEBUG awx.main.dispatch task fe70848e-0573-4e64-829e-15726fdf255d is finished RESULT 2 OKREADY 2021-04-08 14:43:06,142 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(938761af-5cc6-4557-abd0-ccfc9f4453b4, queue=awx_private_queue) [2021-04-08 14:43:06,142: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(938761af-5cc6-4557-abd0-ccfc9f4453b4, queue=awx_private_queue) 2021-04-08 14:43:06,159 DEBUG awx.main.dispatch delivered 938761af-5cc6-4557-abd0-ccfc9f4453b4 to worker[177] qsize 0 2021-04-08 14:43:06,162 DEBUG awx.main.dispatch task 938761af-5cc6-4557-abd0-ccfc9f4453b4 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:43:06,175 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:43:06,178 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:42:36.367643+00:00 2021-04-08 14:43:06,245 DEBUG awx.main.dispatch task 938761af-5cc6-4557-abd0-ccfc9f4453b4 is finished 2021-04-08 14:43:16,202 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d6c19b0a-3c5a-4aaf-a134-25c209266d6c, queue=awx_private_queue) [2021-04-08 14:43:16,202: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d6c19b0a-3c5a-4aaf-a134-25c209266d6c, queue=awx_private_queue) 2021-04-08 14:43:16,216 DEBUG awx.main.dispatch delivered d6c19b0a-3c5a-4aaf-a134-25c209266d6c to worker[180] qsize 0 2021-04-08 14:43:16,218 DEBUG awx.main.dispatch task d6c19b0a-3c5a-4aaf-a134-25c209266d6c starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:43:16,219 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:43:16,237 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:43:16,279 DEBUG awx.main.dispatch task d6c19b0a-3c5a-4aaf-a134-25c209266d6c is finished 2021-04-08 14:43:35,884 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(0856323a-1208-42ba-a49a-955cbc07e4f6, queue=dev-awx-0) [2021-04-08 14:43:35,884: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(0856323a-1208-42ba-a49a-955cbc07e4f6, queue=dev-awx-0) 2021-04-08 14:43:35,929 DEBUG awx.main.dispatch delivered 0856323a-1208-42ba-a49a-955cbc07e4f6 to worker[180] qsize 0 2021-04-08 14:43:35,934 DEBUG awx.main.dispatch task 0856323a-1208-42ba-a49a-955cbc07e4f6 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:43:35,939 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:43:35,999 DEBUG awx.main.dispatch task 0856323a-1208-42ba-a49a-955cbc07e4f6 is finished 2021-04-08 14:43:36,106 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(e1daa709-a80e-4a6f-acb0-8e2e9406b5de, queue=dev-awx-0) [2021-04-08 14:43:36,106: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(e1daa709-a80e-4a6f-acb0-8e2e9406b5de, queue=dev-awx-0) 2021-04-08 14:43:36,140 DEBUG awx.main.dispatch delivered e1daa709-a80e-4a6f-acb0-8e2e9406b5de to worker[177] qsize 0 2021-04-08 14:43:36,142 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f6fcaf48-fb53-47a5-8072-85707bb87692, queue=awx_private_queue) [2021-04-08 14:43:36,142: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f6fcaf48-fb53-47a5-8072-85707bb87692, queue=awx_private_queue) 2021-04-08 14:43:36,144 DEBUG awx.main.dispatch task e1daa709-a80e-4a6f-acb0-8e2e9406b5de starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:43:36,167 DEBUG awx.main.dispatch task e1daa709-a80e-4a6f-acb0-8e2e9406b5de is finished 2021-04-08 14:43:36,202 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(756cda67-6e80-474a-9f58-bc8f21866df9, queue=awx_private_queue) [2021-04-08 14:43:36,202: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(756cda67-6e80-474a-9f58-bc8f21866df9, queue=awx_private_queue) 2021-04-08 14:43:36,160 DEBUG awx.main.dispatch delivered f6fcaf48-fb53-47a5-8072-85707bb87692 to worker[180] qsize 0 2021-04-08 14:43:36,328 DEBUG awx.main.dispatch delivered 756cda67-6e80-474a-9f58-bc8f21866df9 to worker[178] qsize 0 2021-04-08 14:43:36,328 DEBUG awx.main.dispatch task f6fcaf48-fb53-47a5-8072-85707bb87692 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:43:36,332 DEBUG awx.main.dispatch task 756cda67-6e80-474a-9f58-bc8f21866df9 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:43:36,336 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:43:36,343 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:43:36,346 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:43:06.175176+00:00 2021-04-08 14:43:36,362 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:43:36,366 DEBUG awx.main.dispatch task f6fcaf48-fb53-47a5-8072-85707bb87692 is finished 2021-04-08 14:43:36,396 DEBUG awx.main.dispatch task 756cda67-6e80-474a-9f58-bc8f21866df9 is finished 2021-04-08 14:43:56,222 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cdce7c08-1168-4042-8351-afd42e208f33, queue=awx_private_queue) [2021-04-08 14:43:56,222: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cdce7c08-1168-4042-8351-afd42e208f33, queue=awx_private_queue) 2021-04-08 14:43:56,251 DEBUG awx.main.dispatch delivered cdce7c08-1168-4042-8351-afd42e208f33 to worker[176] qsize 0 2021-04-08 14:43:56,261 DEBUG awx.main.dispatch task cdce7c08-1168-4042-8351-afd42e208f33 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:43:56,269 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:43:56,301 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:43:56,337 DEBUG awx.main.dispatch task cdce7c08-1168-4042-8351-afd42e208f33 is finished RESULT 2 OKREADY 2021-04-08 14:44:06,142 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ddad3b18-a044-42c0-a9bc-bb02b06804a3, queue=awx_private_queue) [2021-04-08 14:44:06,142: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ddad3b18-a044-42c0-a9bc-bb02b06804a3, queue=awx_private_queue) 2021-04-08 14:44:06,159 DEBUG awx.main.dispatch delivered ddad3b18-a044-42c0-a9bc-bb02b06804a3 to worker[177] qsize 0 2021-04-08 14:44:06,165 DEBUG awx.main.dispatch task ddad3b18-a044-42c0-a9bc-bb02b06804a3 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:44:06,180 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:44:06,185 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:43:36.343540+00:00 2021-04-08 14:44:06,263 DEBUG awx.main.dispatch task ddad3b18-a044-42c0-a9bc-bb02b06804a3 is finished 2021-04-08 14:44:16,220 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(257d9708-d299-4e46-a912-f8e74467d691, queue=awx_private_queue) [2021-04-08 14:44:16,220: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(257d9708-d299-4e46-a912-f8e74467d691, queue=awx_private_queue) 2021-04-08 14:44:16,236 DEBUG awx.main.dispatch delivered 257d9708-d299-4e46-a912-f8e74467d691 to worker[180] qsize 0 2021-04-08 14:44:16,240 DEBUG awx.main.dispatch task 257d9708-d299-4e46-a912-f8e74467d691 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:44:16,244 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:44:16,273 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:44:16,315 DEBUG awx.main.dispatch task 257d9708-d299-4e46-a912-f8e74467d691 is finished 2021-04-08 14:44:35,889 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(47a0b62c-a77a-4f37-b585-f898c6702d7e, queue=dev-awx-0) [2021-04-08 14:44:35,889: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(47a0b62c-a77a-4f37-b585-f898c6702d7e, queue=dev-awx-0) 2021-04-08 14:44:35,923 DEBUG awx.main.dispatch delivered 47a0b62c-a77a-4f37-b585-f898c6702d7e to worker[178] qsize 0 2021-04-08 14:44:35,927 DEBUG awx.main.dispatch task 47a0b62c-a77a-4f37-b585-f898c6702d7e starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:44:35,931 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:44:35,980 DEBUG awx.main.dispatch task 47a0b62c-a77a-4f37-b585-f898c6702d7e is finished 2021-04-08 14:44:36,106 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ca3fc27a-63c1-4c30-af80-392786fc4db5, queue=dev-awx-0) [2021-04-08 14:44:36,106: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ca3fc27a-63c1-4c30-af80-392786fc4db5, queue=dev-awx-0) 2021-04-08 14:44:36,120 DEBUG awx.main.dispatch delivered ca3fc27a-63c1-4c30-af80-392786fc4db5 to worker[180] qsize 0 2021-04-08 14:44:36,122 DEBUG awx.main.dispatch task ca3fc27a-63c1-4c30-af80-392786fc4db5 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:44:36,135 DEBUG awx.main.dispatch task ca3fc27a-63c1-4c30-af80-392786fc4db5 is finished 2021-04-08 14:44:36,142 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(83691d1f-b377-4d78-b304-293ad367c804, queue=awx_private_queue) [2021-04-08 14:44:36,142: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(83691d1f-b377-4d78-b304-293ad367c804, queue=awx_private_queue) 2021-04-08 14:44:36,156 DEBUG awx.main.dispatch delivered 83691d1f-b377-4d78-b304-293ad367c804 to worker[178] qsize 0 2021-04-08 14:44:36,158 DEBUG awx.main.dispatch task 83691d1f-b377-4d78-b304-293ad367c804 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:44:36,167 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:44:36,169 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:44:06.180755+00:00 2021-04-08 14:44:36,182 DEBUG awx.main.dispatch task 83691d1f-b377-4d78-b304-293ad367c804 is finished 2021-04-08 14:44:36,219 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c0902926-d771-44c7-8fc4-16ff9c65ad63, queue=awx_private_queue) [2021-04-08 14:44:36,219: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c0902926-d771-44c7-8fc4-16ff9c65ad63, queue=awx_private_queue) 2021-04-08 14:44:36,243 DEBUG awx.main.dispatch delivered c0902926-d771-44c7-8fc4-16ff9c65ad63 to worker[178] qsize 0 2021-04-08 14:44:36,414 DEBUG awx.main.dispatch task c0902926-d771-44c7-8fc4-16ff9c65ad63 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:44:36,416 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:44:36,431 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:44:36,471 DEBUG awx.main.dispatch task c0902926-d771-44c7-8fc4-16ff9c65ad63 is finished 2021-04-08 14:44:56,230 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(13acbc00-8e14-4eb0-abb8-3aae88774f3f, queue=awx_private_queue) [2021-04-08 14:44:56,230: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(13acbc00-8e14-4eb0-abb8-3aae88774f3f, queue=awx_private_queue) 2021-04-08 14:44:56,245 DEBUG awx.main.dispatch delivered 13acbc00-8e14-4eb0-abb8-3aae88774f3f to worker[178] qsize 0 2021-04-08 14:44:56,247 DEBUG awx.main.dispatch task 13acbc00-8e14-4eb0-abb8-3aae88774f3f starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:44:56,248 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:44:56,266 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:44:56,301 DEBUG awx.main.dispatch task 13acbc00-8e14-4eb0-abb8-3aae88774f3f is finished RESULT 2 OKREADY 2021-04-08 14:45:06,142 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6879c88f-b249-4f2a-97d9-799d031eaa1d, queue=awx_private_queue) [2021-04-08 14:45:06,142: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6879c88f-b249-4f2a-97d9-799d031eaa1d, queue=awx_private_queue) 2021-04-08 14:45:06,158 DEBUG awx.main.dispatch delivered 6879c88f-b249-4f2a-97d9-799d031eaa1d to worker[176] qsize 0 2021-04-08 14:45:06,160 DEBUG awx.main.dispatch task 6879c88f-b249-4f2a-97d9-799d031eaa1d starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:45:06,181 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:45:06,186 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:44:36.167239+00:00 2021-04-08 14:45:06,236 DEBUG awx.main.dispatch task 6879c88f-b249-4f2a-97d9-799d031eaa1d is finished 2021-04-08 14:45:16,230 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1dd0c823-a7bb-4353-85d4-935a2b2f1801, queue=awx_private_queue) [2021-04-08 14:45:16,230: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1dd0c823-a7bb-4353-85d4-935a2b2f1801, queue=awx_private_queue) 2021-04-08 14:45:16,245 DEBUG awx.main.dispatch delivered 1dd0c823-a7bb-4353-85d4-935a2b2f1801 to worker[180] qsize 0 2021-04-08 14:45:16,247 DEBUG awx.main.dispatch task 1dd0c823-a7bb-4353-85d4-935a2b2f1801 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:45:16,249 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:45:16,265 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:45:16,292 DEBUG awx.main.dispatch task 1dd0c823-a7bb-4353-85d4-935a2b2f1801 is finished 2021-04-08 14:45:35,901 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(0d1139d9-466d-40cf-b342-07000df539d2, queue=dev-awx-0) [2021-04-08 14:45:35,901: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(0d1139d9-466d-40cf-b342-07000df539d2, queue=dev-awx-0) 2021-04-08 14:45:36,145 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(92ae1950-56e7-4583-998b-fdcbd058b889, queue=dev-awx-0) [2021-04-08 14:45:36,145: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(92ae1950-56e7-4583-998b-fdcbd058b889, queue=dev-awx-0) 2021-04-08 14:45:36,164 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(73a5ecc9-58f2-4413-b1c5-0dce9691bff0, queue=awx_private_queue) [2021-04-08 14:45:36,164: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(73a5ecc9-58f2-4413-b1c5-0dce9691bff0, queue=awx_private_queue) 2021-04-08 14:45:36,166 DEBUG awx.main.dispatch delivered 0d1139d9-466d-40cf-b342-07000df539d2 to worker[177] qsize 0 2021-04-08 14:45:36,169 DEBUG awx.main.dispatch delivered 92ae1950-56e7-4583-998b-fdcbd058b889 to worker[180] qsize 0 2021-04-08 14:45:36,176 DEBUG awx.main.dispatch task 92ae1950-56e7-4583-998b-fdcbd058b889 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:45:36,171 DEBUG awx.main.dispatch task 0d1139d9-466d-40cf-b342-07000df539d2 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:45:36,182 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:45:36,185 DEBUG awx.main.dispatch delivered 73a5ecc9-58f2-4413-b1c5-0dce9691bff0 to worker[176] qsize 0 2021-04-08 14:45:36,189 DEBUG awx.main.dispatch task 73a5ecc9-58f2-4413-b1c5-0dce9691bff0 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:45:36,201 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:45:36,203 DEBUG awx.main.dispatch task 92ae1950-56e7-4583-998b-fdcbd058b889 is finished 2021-04-08 14:45:36,205 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:45:06.181526+00:00 2021-04-08 14:45:36,230 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3c347d31-c353-4154-9c95-93f6b4e0c8f3, queue=awx_private_queue) [2021-04-08 14:45:36,230: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3c347d31-c353-4154-9c95-93f6b4e0c8f3, queue=awx_private_queue) 2021-04-08 14:45:36,245 DEBUG awx.main.dispatch delivered 3c347d31-c353-4154-9c95-93f6b4e0c8f3 to worker[180] qsize 0 2021-04-08 14:45:36,249 DEBUG awx.main.dispatch task 3c347d31-c353-4154-9c95-93f6b4e0c8f3 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:45:36,251 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:45:36,252 DEBUG awx.main.dispatch task 0d1139d9-466d-40cf-b342-07000df539d2 is finished 2021-04-08 14:45:36,266 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:45:36,265 DEBUG awx.main.dispatch task 73a5ecc9-58f2-4413-b1c5-0dce9691bff0 is finished 2021-04-08 14:45:36,293 DEBUG awx.main.dispatch task 3c347d31-c353-4154-9c95-93f6b4e0c8f3 is finished 2021-04-08 14:45:56,235 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(af0ecb25-0c96-450a-9f85-25352c690bc0, queue=awx_private_queue) [2021-04-08 14:45:56,235: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(af0ecb25-0c96-450a-9f85-25352c690bc0, queue=awx_private_queue) 2021-04-08 14:45:56,252 DEBUG awx.main.dispatch delivered af0ecb25-0c96-450a-9f85-25352c690bc0 to worker[177] qsize 0 2021-04-08 14:45:56,255 DEBUG awx.main.dispatch task af0ecb25-0c96-450a-9f85-25352c690bc0 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:45:56,256 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:45:56,273 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:45:56,302 DEBUG awx.main.dispatch task af0ecb25-0c96-450a-9f85-25352c690bc0 is finished RESULT 2 OKREADY 2021-04-08 14:46:06,165 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(99913403-4a21-4426-819c-7cc0757c7226, queue=awx_private_queue) [2021-04-08 14:46:06,165: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(99913403-4a21-4426-819c-7cc0757c7226, queue=awx_private_queue) 2021-04-08 14:46:06,184 DEBUG awx.main.dispatch delivered 99913403-4a21-4426-819c-7cc0757c7226 to worker[180] qsize 0 2021-04-08 14:46:06,187 DEBUG awx.main.dispatch task 99913403-4a21-4426-819c-7cc0757c7226 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:46:06,202 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:46:06,205 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:45:36.201804+00:00 2021-04-08 14:46:06,319 DEBUG awx.main.dispatch task 99913403-4a21-4426-819c-7cc0757c7226 is finished 2021-04-08 14:46:16,235 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(055b2d2d-5a69-49fe-b583-83f35fb4d11e, queue=awx_private_queue) [2021-04-08 14:46:16,235: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(055b2d2d-5a69-49fe-b583-83f35fb4d11e, queue=awx_private_queue) 2021-04-08 14:46:16,250 DEBUG awx.main.dispatch delivered 055b2d2d-5a69-49fe-b583-83f35fb4d11e to worker[177] qsize 0 2021-04-08 14:46:16,253 DEBUG awx.main.dispatch task 055b2d2d-5a69-49fe-b583-83f35fb4d11e starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:46:16,254 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:46:16,287 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:46:16,319 DEBUG awx.main.dispatch task 055b2d2d-5a69-49fe-b583-83f35fb4d11e is finished 2021-04-08 14:46:35,903 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b8cf2461-4dbd-4244-a505-435615309131, queue=dev-awx-0) [2021-04-08 14:46:35,903: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b8cf2461-4dbd-4244-a505-435615309131, queue=dev-awx-0) 2021-04-08 14:46:36,122 DEBUG awx.main.dispatch delivered b8cf2461-4dbd-4244-a505-435615309131 to worker[177] qsize 0 2021-04-08 14:46:36,128 DEBUG awx.main.dispatch task b8cf2461-4dbd-4244-a505-435615309131 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:46:36,129 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:46:36,146 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(62142c46-600d-45b9-974d-9720d04fae51, queue=dev-awx-0) [2021-04-08 14:46:36,146: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(62142c46-600d-45b9-974d-9720d04fae51, queue=dev-awx-0) 2021-04-08 14:46:36,167 DEBUG awx.main.dispatch delivered 62142c46-600d-45b9-974d-9720d04fae51 to worker[180] qsize 0 2021-04-08 14:46:36,169 DEBUG awx.main.dispatch task 62142c46-600d-45b9-974d-9720d04fae51 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:46:36,170 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d4753585-ed12-44e2-88e1-42166ae75a29, queue=awx_private_queue) [2021-04-08 14:46:36,170: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d4753585-ed12-44e2-88e1-42166ae75a29, queue=awx_private_queue) 2021-04-08 14:46:36,183 DEBUG awx.main.dispatch task 62142c46-600d-45b9-974d-9720d04fae51 is finished 2021-04-08 14:46:36,184 DEBUG awx.main.dispatch delivered d4753585-ed12-44e2-88e1-42166ae75a29 to worker[178] qsize 0 2021-04-08 14:46:36,185 DEBUG awx.main.dispatch task b8cf2461-4dbd-4244-a505-435615309131 is finished 2021-04-08 14:46:36,188 DEBUG awx.main.dispatch task d4753585-ed12-44e2-88e1-42166ae75a29 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:46:36,203 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:46:36,208 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:46:06.202375+00:00 2021-04-08 14:46:36,232 DEBUG awx.main.dispatch task d4753585-ed12-44e2-88e1-42166ae75a29 is finished 2021-04-08 14:46:36,235 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f92cb819-9748-43ad-a8ff-ccdbd75fd518, queue=awx_private_queue) [2021-04-08 14:46:36,235: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f92cb819-9748-43ad-a8ff-ccdbd75fd518, queue=awx_private_queue) 2021-04-08 14:46:36,251 DEBUG awx.main.dispatch delivered f92cb819-9748-43ad-a8ff-ccdbd75fd518 to worker[178] qsize 0 2021-04-08 14:46:36,254 DEBUG awx.main.dispatch task f92cb819-9748-43ad-a8ff-ccdbd75fd518 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:46:36,256 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:46:36,273 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:46:36,302 DEBUG awx.main.dispatch task f92cb819-9748-43ad-a8ff-ccdbd75fd518 is finished 2021-04-08 14:46:56,257 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(81b7b6f3-aba9-48fc-9697-926889d7d91b, queue=awx_private_queue) [2021-04-08 14:46:56,257: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(81b7b6f3-aba9-48fc-9697-926889d7d91b, queue=awx_private_queue) 2021-04-08 14:46:56,284 DEBUG awx.main.dispatch delivered 81b7b6f3-aba9-48fc-9697-926889d7d91b to worker[178] qsize 0 2021-04-08 14:46:56,291 DEBUG awx.main.dispatch task 81b7b6f3-aba9-48fc-9697-926889d7d91b starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:46:56,294 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:46:56,318 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:46:56,350 DEBUG awx.main.dispatch task 81b7b6f3-aba9-48fc-9697-926889d7d91b is finished RESULT 2 OKREADY 2021-04-08 14:47:06,170 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(81b83d3f-5ee3-48f4-b83b-f1c55b72000c, queue=awx_private_queue) [2021-04-08 14:47:06,170: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(81b83d3f-5ee3-48f4-b83b-f1c55b72000c, queue=awx_private_queue) 2021-04-08 14:47:06,185 DEBUG awx.main.dispatch delivered 81b83d3f-5ee3-48f4-b83b-f1c55b72000c to worker[176] qsize 0 2021-04-08 14:47:06,189 DEBUG awx.main.dispatch task 81b83d3f-5ee3-48f4-b83b-f1c55b72000c starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:47:06,214 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:47:06,220 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:46:36.204001+00:00 2021-04-08 14:47:06,290 DEBUG awx.main.dispatch task 81b83d3f-5ee3-48f4-b83b-f1c55b72000c is finished 2021-04-08 14:47:16,251 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(253bcec0-702e-4792-8eee-77dcce534921, queue=awx_private_queue) [2021-04-08 14:47:16,251: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(253bcec0-702e-4792-8eee-77dcce534921, queue=awx_private_queue) 2021-04-08 14:47:16,267 DEBUG awx.main.dispatch delivered 253bcec0-702e-4792-8eee-77dcce534921 to worker[176] qsize 0 2021-04-08 14:47:16,269 DEBUG awx.main.dispatch task 253bcec0-702e-4792-8eee-77dcce534921 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:47:16,271 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:47:16,739 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:47:16,783 DEBUG awx.main.dispatch task 253bcec0-702e-4792-8eee-77dcce534921 is finished 2021-04-08 14:47:35,883 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(63f5f835-c474-46bc-89a7-96aed6ccb564, queue=dev-awx-0) [2021-04-08 14:47:35,883: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(63f5f835-c474-46bc-89a7-96aed6ccb564, queue=dev-awx-0) 2021-04-08 14:47:35,955 DEBUG awx.main.dispatch delivered 63f5f835-c474-46bc-89a7-96aed6ccb564 to worker[178] qsize 0 2021-04-08 14:47:35,962 DEBUG awx.main.dispatch task 63f5f835-c474-46bc-89a7-96aed6ccb564 starting awx.main.tasks.awx_isolated_heartbeat(*[]) 2021-04-08 14:47:35,968 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks. 2021-04-08 14:47:36,012 DEBUG awx.main.dispatch task 63f5f835-c474-46bc-89a7-96aed6ccb564 is finished 2021-04-08 14:47:35,955 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(34a8f9e3-8666-4901-85cd-07979ec79e95, queue=dev-awx-0) [2021-04-08 14:47:35,955: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(34a8f9e3-8666-4901-85cd-07979ec79e95, queue=dev-awx-0) 2021-04-08 14:47:36,186 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(30fb2650-9310-43e3-9709-6483c6ba417f, queue=dev-awx-0) [2021-04-08 14:47:36,186: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(30fb2650-9310-43e3-9709-6483c6ba417f, queue=dev-awx-0) 2021-04-08 14:47:36,203 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a057734a-ce00-43dd-aa6c-7125d61baade, queue=awx_private_queue) [2021-04-08 14:47:36,203: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a057734a-ce00-43dd-aa6c-7125d61baade, queue=awx_private_queue) 2021-04-08 14:47:36,251 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(82f79358-f97a-4ae2-8e1f-230c29ca20ca, queue=awx_private_queue) [2021-04-08 14:47:36,251: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(82f79358-f97a-4ae2-8e1f-230c29ca20ca, queue=awx_private_queue) 2021-04-08 14:47:36,270 DEBUG awx.main.dispatch delivered 34a8f9e3-8666-4901-85cd-07979ec79e95 to worker[177] qsize 0 2021-04-08 14:47:36,274 DEBUG awx.main.dispatch delivered 30fb2650-9310-43e3-9709-6483c6ba417f to worker[176] qsize 0 2021-04-08 14:47:36,276 DEBUG awx.main.dispatch delivered a057734a-ce00-43dd-aa6c-7125d61baade to worker[180] qsize 0 2021-04-08 14:47:36,278 DEBUG awx.main.dispatch delivered 82f79358-f97a-4ae2-8e1f-230c29ca20ca to worker[178] qsize 0 2021-04-08 14:47:36,275 DEBUG awx.main.dispatch task 34a8f9e3-8666-4901-85cd-07979ec79e95 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:47:36,279 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:47:36,277 DEBUG awx.main.dispatch task 30fb2650-9310-43e3-9709-6483c6ba417f starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:47:36,279 DEBUG awx.main.dispatch task a057734a-ce00-43dd-aa6c-7125d61baade starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:47:36,286 DEBUG awx.main.dispatch task 82f79358-f97a-4ae2-8e1f-230c29ca20ca starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:47:36,288 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:47:36,302 DEBUG awx.main.dispatch task 30fb2650-9310-43e3-9709-6483c6ba417f is finished 2021-04-08 14:47:36,307 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:47:36,314 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:47:06.214639+00:00 2021-04-08 14:47:36,319 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:47:36,373 DEBUG awx.main.dispatch task 82f79358-f97a-4ae2-8e1f-230c29ca20ca is finished 2021-04-08 14:47:36,420 DEBUG awx.main.dispatch task 34a8f9e3-8666-4901-85cd-07979ec79e95 is finished 2021-04-08 14:47:36,446 DEBUG awx.main.dispatch task a057734a-ce00-43dd-aa6c-7125d61baade is finished 2021-04-08 14:47:56,262 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7266a429-868a-44b6-8dbd-dfc9e189a65e, queue=awx_private_queue) [2021-04-08 14:47:56,262: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7266a429-868a-44b6-8dbd-dfc9e189a65e, queue=awx_private_queue) 2021-04-08 14:47:56,279 DEBUG awx.main.dispatch delivered 7266a429-868a-44b6-8dbd-dfc9e189a65e to worker[176] qsize 0 2021-04-08 14:47:56,282 DEBUG awx.main.dispatch task 7266a429-868a-44b6-8dbd-dfc9e189a65e starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:47:56,284 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:47:56,382 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:47:56,453 DEBUG awx.main.dispatch task 7266a429-868a-44b6-8dbd-dfc9e189a65e is finished RESULT 2 OKREADY 2021-04-08 14:48:06,203 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4067b656-26fc-4328-a8ca-1498a41e2567, queue=awx_private_queue) [2021-04-08 14:48:06,203: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4067b656-26fc-4328-a8ca-1498a41e2567, queue=awx_private_queue) 2021-04-08 14:48:06,219 DEBUG awx.main.dispatch delivered 4067b656-26fc-4328-a8ca-1498a41e2567 to worker[177] qsize 0 2021-04-08 14:48:06,222 DEBUG awx.main.dispatch task 4067b656-26fc-4328-a8ca-1498a41e2567 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:48:06,234 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:48:06,241 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:47:36.307813+00:00 2021-04-08 14:48:06,271 DEBUG awx.main.dispatch task 4067b656-26fc-4328-a8ca-1498a41e2567 is finished 2021-04-08 14:48:16,269 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(81ebe2f0-1168-408e-bfcb-594df647b7e7, queue=awx_private_queue) [2021-04-08 14:48:16,269: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(81ebe2f0-1168-408e-bfcb-594df647b7e7, queue=awx_private_queue) 2021-04-08 14:48:16,293 DEBUG awx.main.dispatch delivered 81ebe2f0-1168-408e-bfcb-594df647b7e7 to worker[176] qsize 0 2021-04-08 14:48:16,302 DEBUG awx.main.dispatch task 81ebe2f0-1168-408e-bfcb-594df647b7e7 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:48:16,308 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:48:16,335 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:48:16,375 DEBUG awx.main.dispatch task 81ebe2f0-1168-408e-bfcb-594df647b7e7 is finished 2021-04-08 14:48:35,962 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(deac2821-a2f1-4e37-9e89-9de6219edf04, queue=dev-awx-0) [2021-04-08 14:48:35,962: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(deac2821-a2f1-4e37-9e89-9de6219edf04, queue=dev-awx-0) 2021-04-08 14:48:36,183 DEBUG awx.main.dispatch delivered deac2821-a2f1-4e37-9e89-9de6219edf04 to worker[178] qsize 0 2021-04-08 14:48:36,187 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(0f523e19-d99e-42a1-b08b-513874363b1b, queue=dev-awx-0) [2021-04-08 14:48:36,187: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(0f523e19-d99e-42a1-b08b-513874363b1b, queue=dev-awx-0) 2021-04-08 14:48:36,186 DEBUG awx.main.dispatch task deac2821-a2f1-4e37-9e89-9de6219edf04 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:48:36,191 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:48:36,209 DEBUG awx.main.dispatch delivered 0f523e19-d99e-42a1-b08b-513874363b1b to worker[180] qsize 0 2021-04-08 14:48:36,211 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(12318115-054b-424d-9f21-df31f3ddd69b, queue=awx_private_queue) [2021-04-08 14:48:36,211: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(12318115-054b-424d-9f21-df31f3ddd69b, queue=awx_private_queue) 2021-04-08 14:48:36,213 DEBUG awx.main.dispatch task 0f523e19-d99e-42a1-b08b-513874363b1b starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:48:36,229 DEBUG awx.main.dispatch task 0f523e19-d99e-42a1-b08b-513874363b1b is finished 2021-04-08 14:48:36,230 DEBUG awx.main.dispatch delivered 12318115-054b-424d-9f21-df31f3ddd69b to worker[177] qsize 0 2021-04-08 14:48:36,234 DEBUG awx.main.dispatch task 12318115-054b-424d-9f21-df31f3ddd69b starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:48:36,247 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:48:36,252 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:48:06.235142+00:00 2021-04-08 14:48:36,264 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2fe75645-d444-418e-a4b7-139178479b37, queue=awx_private_queue) [2021-04-08 14:48:36,264: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2fe75645-d444-418e-a4b7-139178479b37, queue=awx_private_queue) 2021-04-08 14:48:36,280 DEBUG awx.main.dispatch task deac2821-a2f1-4e37-9e89-9de6219edf04 is finished 2021-04-08 14:48:36,281 DEBUG awx.main.dispatch delivered 2fe75645-d444-418e-a4b7-139178479b37 to worker[180] qsize 0 2021-04-08 14:48:36,283 DEBUG awx.main.dispatch task 2fe75645-d444-418e-a4b7-139178479b37 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:48:36,285 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:48:36,302 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:48:36,303 DEBUG awx.main.dispatch task 12318115-054b-424d-9f21-df31f3ddd69b is finished 2021-04-08 14:48:36,332 DEBUG awx.main.dispatch task 2fe75645-d444-418e-a4b7-139178479b37 is finished 2021-04-08 14:48:56,276 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7f2aa0eb-a183-49ad-a9d8-695a2b7b70d9, queue=awx_private_queue) [2021-04-08 14:48:56,276: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7f2aa0eb-a183-49ad-a9d8-695a2b7b70d9, queue=awx_private_queue) 2021-04-08 14:48:56,295 DEBUG awx.main.dispatch delivered 7f2aa0eb-a183-49ad-a9d8-695a2b7b70d9 to worker[176] qsize 0 2021-04-08 14:48:56,302 DEBUG awx.main.dispatch task 7f2aa0eb-a183-49ad-a9d8-695a2b7b70d9 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:48:56,306 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:48:56,333 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:48:56,365 DEBUG awx.main.dispatch task 7f2aa0eb-a183-49ad-a9d8-695a2b7b70d9 is finished RESULT 2 OKREADY 2021-04-08 14:49:06,211 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b465433f-2ff5-4400-a93e-ffa1c23a01df, queue=awx_private_queue) [2021-04-08 14:49:06,211: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b465433f-2ff5-4400-a93e-ffa1c23a01df, queue=awx_private_queue) 2021-04-08 14:49:06,227 DEBUG awx.main.dispatch delivered b465433f-2ff5-4400-a93e-ffa1c23a01df to worker[177] qsize 0 2021-04-08 14:49:06,232 DEBUG awx.main.dispatch task b465433f-2ff5-4400-a93e-ffa1c23a01df starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:49:06,248 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:49:06,255 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:48:36.247508+00:00 2021-04-08 14:49:06,311 DEBUG awx.main.dispatch task b465433f-2ff5-4400-a93e-ffa1c23a01df is finished 2021-04-08 14:49:16,275 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ade56914-f9f5-484f-bc4b-fb0bb418b355, queue=awx_private_queue) [2021-04-08 14:49:16,275: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ade56914-f9f5-484f-bc4b-fb0bb418b355, queue=awx_private_queue) 2021-04-08 14:49:16,292 DEBUG awx.main.dispatch delivered ade56914-f9f5-484f-bc4b-fb0bb418b355 to worker[176] qsize 0 2021-04-08 14:49:16,295 DEBUG awx.main.dispatch task ade56914-f9f5-484f-bc4b-fb0bb418b355 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:49:16,296 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:49:16,314 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:49:16,345 DEBUG awx.main.dispatch task ade56914-f9f5-484f-bc4b-fb0bb418b355 is finished 2021-04-08 14:49:35,971 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5e69355d-c767-4486-8b97-0506fe73ea62, queue=dev-awx-0) [2021-04-08 14:49:35,971: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5e69355d-c767-4486-8b97-0506fe73ea62, queue=dev-awx-0) 2021-04-08 14:49:36,243 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d0470e43-4ce0-4018-911c-6e98def91d68, queue=dev-awx-0) [2021-04-08 14:49:36,243: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d0470e43-4ce0-4018-911c-6e98def91d68, queue=dev-awx-0) 2021-04-08 14:49:36,256 DEBUG awx.main.dispatch delivered 5e69355d-c767-4486-8b97-0506fe73ea62 to worker[180] qsize 0 2021-04-08 14:49:36,262 DEBUG awx.main.dispatch task 5e69355d-c767-4486-8b97-0506fe73ea62 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:49:36,267 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:49:36,280 DEBUG awx.main.dispatch delivered d0470e43-4ce0-4018-911c-6e98def91d68 to worker[176] qsize 0 2021-04-08 14:49:36,281 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(32102550-13dc-4c94-9132-1a89be395608, queue=awx_private_queue) [2021-04-08 14:49:36,281: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(32102550-13dc-4c94-9132-1a89be395608, queue=awx_private_queue) 2021-04-08 14:49:36,283 DEBUG awx.main.dispatch task d0470e43-4ce0-4018-911c-6e98def91d68 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:49:36,294 DEBUG awx.main.dispatch task d0470e43-4ce0-4018-911c-6e98def91d68 is finished 2021-04-08 14:49:36,298 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2382b63b-d626-44fd-8488-fee5dfe0a832, queue=awx_private_queue) [2021-04-08 14:49:36,298: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2382b63b-d626-44fd-8488-fee5dfe0a832, queue=awx_private_queue) 2021-04-08 14:49:36,298 DEBUG awx.main.dispatch delivered 32102550-13dc-4c94-9132-1a89be395608 to worker[176] qsize 0 2021-04-08 14:49:36,304 DEBUG awx.main.dispatch task 32102550-13dc-4c94-9132-1a89be395608 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:49:36,312 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:49:36,312 DEBUG awx.main.dispatch delivered 2382b63b-d626-44fd-8488-fee5dfe0a832 to worker[178] qsize 0 2021-04-08 14:49:36,315 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:49:06.248279+00:00 2021-04-08 14:49:36,317 DEBUG awx.main.dispatch task 2382b63b-d626-44fd-8488-fee5dfe0a832 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:49:36,322 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:49:36,341 DEBUG awx.main.dispatch task 5e69355d-c767-4486-8b97-0506fe73ea62 is finished 2021-04-08 14:49:36,347 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:49:36,354 DEBUG awx.main.dispatch task 32102550-13dc-4c94-9132-1a89be395608 is finished 2021-04-08 14:49:36,378 DEBUG awx.main.dispatch task 2382b63b-d626-44fd-8488-fee5dfe0a832 is finished 2021-04-08 14:49:56,317 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d4432017-0baf-4f9c-9ba2-2925196a6ac9, queue=awx_private_queue) [2021-04-08 14:49:56,317: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d4432017-0baf-4f9c-9ba2-2925196a6ac9, queue=awx_private_queue) 2021-04-08 14:49:56,358 DEBUG awx.main.dispatch delivered d4432017-0baf-4f9c-9ba2-2925196a6ac9 to worker[177] qsize 0 2021-04-08 14:49:56,365 DEBUG awx.main.dispatch task d4432017-0baf-4f9c-9ba2-2925196a6ac9 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:49:56,370 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:49:56,400 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:49:56,435 DEBUG awx.main.dispatch task d4432017-0baf-4f9c-9ba2-2925196a6ac9 is finished RESULT 2 OKREADY 2021-04-08 14:50:06,282 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c27a445d-2f6f-47bc-b94c-7cb65453b509, queue=awx_private_queue) [2021-04-08 14:50:06,282: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c27a445d-2f6f-47bc-b94c-7cb65453b509, queue=awx_private_queue) 2021-04-08 14:50:06,299 DEBUG awx.main.dispatch delivered c27a445d-2f6f-47bc-b94c-7cb65453b509 to worker[177] qsize 0 2021-04-08 14:50:06,302 DEBUG awx.main.dispatch task c27a445d-2f6f-47bc-b94c-7cb65453b509 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:50:06,314 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:50:06,317 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:49:36.312706+00:00 2021-04-08 14:50:06,409 DEBUG awx.main.dispatch task c27a445d-2f6f-47bc-b94c-7cb65453b509 is finished 2021-04-08 14:50:16,315 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(71ccfeec-5582-4f21-92e0-7a45152791e9, queue=awx_private_queue) [2021-04-08 14:50:16,315: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(71ccfeec-5582-4f21-92e0-7a45152791e9, queue=awx_private_queue) 2021-04-08 14:50:16,331 DEBUG awx.main.dispatch delivered 71ccfeec-5582-4f21-92e0-7a45152791e9 to worker[177] qsize 0 2021-04-08 14:50:16,333 DEBUG awx.main.dispatch task 71ccfeec-5582-4f21-92e0-7a45152791e9 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:50:16,334 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:50:16,351 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:50:16,379 DEBUG awx.main.dispatch task 71ccfeec-5582-4f21-92e0-7a45152791e9 is finished 2021-04-08 14:50:35,974 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c56fa7b9-6117-4888-af04-018e00b21efe, queue=dev-awx-0) [2021-04-08 14:50:35,974: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c56fa7b9-6117-4888-af04-018e00b21efe, queue=dev-awx-0) 2021-04-08 14:50:36,230 DEBUG awx.main.dispatch delivered c56fa7b9-6117-4888-af04-018e00b21efe to worker[177] qsize 0 2021-04-08 14:50:36,232 DEBUG awx.main.dispatch task c56fa7b9-6117-4888-af04-018e00b21efe starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:50:36,233 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:50:36,242 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ef625944-e8eb-4348-8008-dbea82fe6661, queue=dev-awx-0) [2021-04-08 14:50:36,242: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ef625944-e8eb-4348-8008-dbea82fe6661, queue=dev-awx-0) 2021-04-08 14:50:36,268 DEBUG awx.main.dispatch delivered ef625944-e8eb-4348-8008-dbea82fe6661 to worker[178] qsize 0 2021-04-08 14:50:36,272 DEBUG awx.main.dispatch task ef625944-e8eb-4348-8008-dbea82fe6661 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:50:36,280 DEBUG awx.main.dispatch task c56fa7b9-6117-4888-af04-018e00b21efe is finished 2021-04-08 14:50:36,282 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(743dce36-4f4f-4e54-bd73-f70984ab5593, queue=awx_private_queue) [2021-04-08 14:50:36,282: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(743dce36-4f4f-4e54-bd73-f70984ab5593, queue=awx_private_queue) 2021-04-08 14:50:36,291 DEBUG awx.main.dispatch task ef625944-e8eb-4348-8008-dbea82fe6661 is finished 2021-04-08 14:50:36,309 DEBUG awx.main.dispatch delivered 743dce36-4f4f-4e54-bd73-f70984ab5593 to worker[180] qsize 0 2021-04-08 14:50:36,314 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fedd47bd-08b6-453e-964c-e19bb7942826, queue=awx_private_queue) [2021-04-08 14:50:36,314: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fedd47bd-08b6-453e-964c-e19bb7942826, queue=awx_private_queue) 2021-04-08 14:50:36,314 DEBUG awx.main.dispatch task 743dce36-4f4f-4e54-bd73-f70984ab5593 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:50:36,327 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:50:36,329 DEBUG awx.main.dispatch delivered fedd47bd-08b6-453e-964c-e19bb7942826 to worker[178] qsize 0 2021-04-08 14:50:36,333 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:50:06.314229+00:00 2021-04-08 14:50:36,332 DEBUG awx.main.dispatch task fedd47bd-08b6-453e-964c-e19bb7942826 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:50:36,334 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:50:36,351 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:50:36,359 DEBUG awx.main.dispatch task 743dce36-4f4f-4e54-bd73-f70984ab5593 is finished 2021-04-08 14:50:36,386 DEBUG awx.main.dispatch task fedd47bd-08b6-453e-964c-e19bb7942826 is finished 2021-04-08 14:50:56,324 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d969bb45-d8ef-4d86-90fc-7afeeed7eeee, queue=awx_private_queue) [2021-04-08 14:50:56,324: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d969bb45-d8ef-4d86-90fc-7afeeed7eeee, queue=awx_private_queue) 2021-04-08 14:50:56,340 DEBUG awx.main.dispatch delivered d969bb45-d8ef-4d86-90fc-7afeeed7eeee to worker[180] qsize 0 2021-04-08 14:50:56,342 DEBUG awx.main.dispatch task d969bb45-d8ef-4d86-90fc-7afeeed7eeee starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:50:56,343 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:50:56,363 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:50:56,394 DEBUG awx.main.dispatch task d969bb45-d8ef-4d86-90fc-7afeeed7eeee is finished RESULT 2 OKREADY 2021-04-08 14:51:06,283 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(41e12151-0dd9-47f9-9884-d1fc3756dd40, queue=awx_private_queue) [2021-04-08 14:51:06,283: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(41e12151-0dd9-47f9-9884-d1fc3756dd40, queue=awx_private_queue) 2021-04-08 14:51:06,297 DEBUG awx.main.dispatch delivered 41e12151-0dd9-47f9-9884-d1fc3756dd40 to worker[178] qsize 0 2021-04-08 14:51:06,299 DEBUG awx.main.dispatch task 41e12151-0dd9-47f9-9884-d1fc3756dd40 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:51:06,308 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:51:06,311 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:50:36.327284+00:00 2021-04-08 14:51:06,404 DEBUG awx.main.dispatch task 41e12151-0dd9-47f9-9884-d1fc3756dd40 is finished 2021-04-08 14:51:16,323 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0cd506eb-e6a7-480c-864e-e9f2323dcbc3, queue=awx_private_queue) [2021-04-08 14:51:16,323: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0cd506eb-e6a7-480c-864e-e9f2323dcbc3, queue=awx_private_queue) 2021-04-08 14:51:16,337 DEBUG awx.main.dispatch delivered 0cd506eb-e6a7-480c-864e-e9f2323dcbc3 to worker[180] qsize 0 2021-04-08 14:51:16,339 DEBUG awx.main.dispatch task 0cd506eb-e6a7-480c-864e-e9f2323dcbc3 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:51:16,340 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:51:16,356 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:51:16,393 DEBUG awx.main.dispatch task 0cd506eb-e6a7-480c-864e-e9f2323dcbc3 is finished 2021-04-08 14:51:35,991 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8e9ba58f-8c58-44fd-a63f-38421b00fe0c, queue=dev-awx-0) [2021-04-08 14:51:35,991: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8e9ba58f-8c58-44fd-a63f-38421b00fe0c, queue=dev-awx-0) 2021-04-08 14:51:36,236 DEBUG awx.main.dispatch delivered 8e9ba58f-8c58-44fd-a63f-38421b00fe0c to worker[180] qsize 0 2021-04-08 14:51:36,242 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(350000c0-e4cb-48a5-9769-e9820a1de1fa, queue=dev-awx-0) [2021-04-08 14:51:36,242: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(350000c0-e4cb-48a5-9769-e9820a1de1fa, queue=dev-awx-0) 2021-04-08 14:51:36,241 DEBUG awx.main.dispatch task 8e9ba58f-8c58-44fd-a63f-38421b00fe0c starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:51:36,245 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:51:36,258 DEBUG awx.main.dispatch delivered 350000c0-e4cb-48a5-9769-e9820a1de1fa to worker[177] qsize 0 2021-04-08 14:51:36,261 DEBUG awx.main.dispatch task 350000c0-e4cb-48a5-9769-e9820a1de1fa starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:51:36,280 DEBUG awx.main.dispatch task 350000c0-e4cb-48a5-9769-e9820a1de1fa is finished 2021-04-08 14:51:36,283 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c3761bf5-8536-4757-a10b-66177d827a8d, queue=awx_private_queue) [2021-04-08 14:51:36,283: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c3761bf5-8536-4757-a10b-66177d827a8d, queue=awx_private_queue) 2021-04-08 14:51:36,302 DEBUG awx.main.dispatch delivered c3761bf5-8536-4757-a10b-66177d827a8d to worker[178] qsize 0 2021-04-08 14:51:36,306 DEBUG awx.main.dispatch task c3761bf5-8536-4757-a10b-66177d827a8d starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:51:36,322 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:51:36,324 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(600c47ea-0edb-4c81-b1e0-73a830e9c1b3, queue=awx_private_queue) 2021-04-08 14:51:36,325 DEBUG awx.main.dispatch task 8e9ba58f-8c58-44fd-a63f-38421b00fe0c is finished [2021-04-08 14:51:36,324: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(600c47ea-0edb-4c81-b1e0-73a830e9c1b3, queue=awx_private_queue) 2021-04-08 14:51:36,328 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:51:06.308265+00:00 2021-04-08 14:51:36,345 DEBUG awx.main.dispatch task c3761bf5-8536-4757-a10b-66177d827a8d is finished 2021-04-08 14:51:36,352 DEBUG awx.main.dispatch delivered 600c47ea-0edb-4c81-b1e0-73a830e9c1b3 to worker[177] qsize 0 2021-04-08 14:51:36,354 DEBUG awx.main.dispatch task 600c47ea-0edb-4c81-b1e0-73a830e9c1b3 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:51:36,356 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:51:36,374 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:51:36,404 DEBUG awx.main.dispatch task 600c47ea-0edb-4c81-b1e0-73a830e9c1b3 is finished 2021-04-08 14:51:56,325 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5f9ea5d6-3f51-44f1-b56c-7b0b589779b3, queue=awx_private_queue) [2021-04-08 14:51:56,325: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5f9ea5d6-3f51-44f1-b56c-7b0b589779b3, queue=awx_private_queue) 2021-04-08 14:51:56,343 DEBUG awx.main.dispatch delivered 5f9ea5d6-3f51-44f1-b56c-7b0b589779b3 to worker[178] qsize 0 2021-04-08 14:51:56,346 DEBUG awx.main.dispatch task 5f9ea5d6-3f51-44f1-b56c-7b0b589779b3 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:51:56,347 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:51:56,369 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:51:56,400 DEBUG awx.main.dispatch task 5f9ea5d6-3f51-44f1-b56c-7b0b589779b3 is finished RESULT 2 OKREADY 2021-04-08 14:52:06,283 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bdb77366-3de6-496d-b37c-810d68e7c352, queue=awx_private_queue) [2021-04-08 14:52:06,283: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bdb77366-3de6-496d-b37c-810d68e7c352, queue=awx_private_queue) 2021-04-08 14:52:06,299 DEBUG awx.main.dispatch delivered bdb77366-3de6-496d-b37c-810d68e7c352 to worker[178] qsize 0 2021-04-08 14:52:06,301 DEBUG awx.main.dispatch task bdb77366-3de6-496d-b37c-810d68e7c352 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:52:06,317 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:52:06,322 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:51:36.322828+00:00 2021-04-08 14:52:06,360 DEBUG awx.main.dispatch task bdb77366-3de6-496d-b37c-810d68e7c352 is finished 2021-04-08 14:52:16,324 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5bf8b526-b774-4b12-b148-9b50a5f3104c, queue=awx_private_queue) [2021-04-08 14:52:16,324: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5bf8b526-b774-4b12-b148-9b50a5f3104c, queue=awx_private_queue) 2021-04-08 14:52:16,347 DEBUG awx.main.dispatch delivered 5bf8b526-b774-4b12-b148-9b50a5f3104c to worker[180] qsize 0 2021-04-08 14:52:16,350 DEBUG awx.main.dispatch task 5bf8b526-b774-4b12-b148-9b50a5f3104c starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:52:16,353 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:52:16,374 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:52:16,408 DEBUG awx.main.dispatch task 5bf8b526-b774-4b12-b148-9b50a5f3104c is finished 2021-04-08 14:52:35,986 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b51c51f2-9a36-4698-a25c-f90a2a84e815, queue=dev-awx-0) [2021-04-08 14:52:35,986: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b51c51f2-9a36-4698-a25c-f90a2a84e815, queue=dev-awx-0) 2021-04-08 14:52:36,032 DEBUG awx.main.dispatch delivered b51c51f2-9a36-4698-a25c-f90a2a84e815 to worker[180] qsize 0 2021-04-08 14:52:36,035 DEBUG awx.main.dispatch task b51c51f2-9a36-4698-a25c-f90a2a84e815 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:52:36,038 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:52:36,090 DEBUG awx.main.dispatch task b51c51f2-9a36-4698-a25c-f90a2a84e815 is finished 2021-04-08 14:52:36,242 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d5dcbf5b-f98a-4454-83d1-059b443c8a65, queue=dev-awx-0) [2021-04-08 14:52:36,242: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d5dcbf5b-f98a-4454-83d1-059b443c8a65, queue=dev-awx-0) 2021-04-08 14:52:36,440 DEBUG awx.main.dispatch delivered d5dcbf5b-f98a-4454-83d1-059b443c8a65 to worker[176] qsize 0 2021-04-08 14:52:36,442 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(13d4a70c-08d4-4005-9025-599324503ded, queue=awx_private_queue) [2021-04-08 14:52:36,442: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(13d4a70c-08d4-4005-9025-599324503ded, queue=awx_private_queue) 2021-04-08 14:52:36,444 DEBUG awx.main.dispatch task d5dcbf5b-f98a-4454-83d1-059b443c8a65 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:52:36,456 DEBUG awx.main.dispatch delivered 13d4a70c-08d4-4005-9025-599324503ded to worker[177] qsize 0 2021-04-08 14:52:36,458 DEBUG awx.main.dispatch task 13d4a70c-08d4-4005-9025-599324503ded starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:52:36,458 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a4170d4e-c083-4f68-bfab-8e422ea25c08, queue=awx_private_queue) [2021-04-08 14:52:36,458: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a4170d4e-c083-4f68-bfab-8e422ea25c08, queue=awx_private_queue) 2021-04-08 14:52:36,463 DEBUG awx.main.dispatch task d5dcbf5b-f98a-4454-83d1-059b443c8a65 is finished 2021-04-08 14:52:36,467 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:52:36,470 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:52:06.317904+00:00 2021-04-08 14:52:36,485 DEBUG awx.main.dispatch delivered a4170d4e-c083-4f68-bfab-8e422ea25c08 to worker[176] qsize 0 2021-04-08 14:52:36,487 DEBUG awx.main.dispatch task a4170d4e-c083-4f68-bfab-8e422ea25c08 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:52:36,488 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:52:36,501 DEBUG awx.main.dispatch task 13d4a70c-08d4-4005-9025-599324503ded is finished 2021-04-08 14:52:36,504 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:52:36,532 DEBUG awx.main.dispatch task a4170d4e-c083-4f68-bfab-8e422ea25c08 is finished 2021-04-08 14:52:56,468 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b84b81b8-26a7-465a-97b1-b299e9423e31, queue=awx_private_queue) [2021-04-08 14:52:56,468: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b84b81b8-26a7-465a-97b1-b299e9423e31, queue=awx_private_queue) 2021-04-08 14:52:56,492 DEBUG awx.main.dispatch delivered b84b81b8-26a7-465a-97b1-b299e9423e31 to worker[180] qsize 0 2021-04-08 14:52:56,494 DEBUG awx.main.dispatch task b84b81b8-26a7-465a-97b1-b299e9423e31 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:52:56,496 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:52:56,519 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:52:56,550 DEBUG awx.main.dispatch task b84b81b8-26a7-465a-97b1-b299e9423e31 is finished RESULT 2 OKREADY 2021-04-08 14:53:06,442 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4c398266-bdc6-4dee-a1b5-cb1d584b49ed, queue=awx_private_queue) [2021-04-08 14:53:06,442: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4c398266-bdc6-4dee-a1b5-cb1d584b49ed, queue=awx_private_queue) 2021-04-08 14:53:06,460 DEBUG awx.main.dispatch delivered 4c398266-bdc6-4dee-a1b5-cb1d584b49ed to worker[177] qsize 0 2021-04-08 14:53:06,463 DEBUG awx.main.dispatch task 4c398266-bdc6-4dee-a1b5-cb1d584b49ed starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:53:06,498 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:53:06,532 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:52:36.468174+00:00 2021-04-08 14:53:06,679 DEBUG awx.main.dispatch task 4c398266-bdc6-4dee-a1b5-cb1d584b49ed is finished 2021-04-08 14:53:16,468 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(104edf23-c65d-4084-8e0d-30f671c132ca, queue=awx_private_queue) [2021-04-08 14:53:16,468: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(104edf23-c65d-4084-8e0d-30f671c132ca, queue=awx_private_queue) 2021-04-08 14:53:16,496 DEBUG awx.main.dispatch delivered 104edf23-c65d-4084-8e0d-30f671c132ca to worker[180] qsize 0 2021-04-08 14:53:16,498 DEBUG awx.main.dispatch task 104edf23-c65d-4084-8e0d-30f671c132ca starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:53:16,500 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:53:16,517 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:53:16,565 DEBUG awx.main.dispatch task 104edf23-c65d-4084-8e0d-30f671c132ca is finished 2021-04-08 14:53:35,986 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a5298350-a352-4bc2-b707-2d4682993169, queue=dev-awx-0) [2021-04-08 14:53:35,986: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a5298350-a352-4bc2-b707-2d4682993169, queue=dev-awx-0) 2021-04-08 14:53:36,164 DEBUG awx.main.dispatch delivered a5298350-a352-4bc2-b707-2d4682993169 to worker[180] qsize 0 2021-04-08 14:53:36,166 DEBUG awx.main.dispatch task a5298350-a352-4bc2-b707-2d4682993169 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:53:36,168 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:53:36,242 DEBUG awx.main.dispatch task a5298350-a352-4bc2-b707-2d4682993169 is finished 2021-04-08 14:53:36,242 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(58c51be9-6cf2-44c4-bf2b-f08791238ef9, queue=dev-awx-0) [2021-04-08 14:53:36,242: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(58c51be9-6cf2-44c4-bf2b-f08791238ef9, queue=dev-awx-0) 2021-04-08 14:53:36,492 DEBUG awx.main.dispatch delivered 58c51be9-6cf2-44c4-bf2b-f08791238ef9 to worker[180] qsize 0 2021-04-08 14:53:36,494 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(87a053df-7697-424b-88f1-67f29706e8f0, queue=awx_private_queue) [2021-04-08 14:53:36,494: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(87a053df-7697-424b-88f1-67f29706e8f0, queue=awx_private_queue) 2021-04-08 14:53:36,495 DEBUG awx.main.dispatch task 58c51be9-6cf2-44c4-bf2b-f08791238ef9 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:53:36,511 DEBUG awx.main.dispatch task 58c51be9-6cf2-44c4-bf2b-f08791238ef9 is finished 2021-04-08 14:53:36,516 DEBUG awx.main.dispatch delivered 87a053df-7697-424b-88f1-67f29706e8f0 to worker[180] qsize 0 2021-04-08 14:53:36,517 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c4c457ca-db7c-4009-930b-37a7ea9e22d5, queue=awx_private_queue) [2021-04-08 14:53:36,517: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c4c457ca-db7c-4009-930b-37a7ea9e22d5, queue=awx_private_queue) 2021-04-08 14:53:36,518 DEBUG awx.main.dispatch task 87a053df-7697-424b-88f1-67f29706e8f0 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:53:36,526 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:53:36,528 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:53:06.499061+00:00 2021-04-08 14:53:36,532 DEBUG awx.main.dispatch delivered c4c457ca-db7c-4009-930b-37a7ea9e22d5 to worker[177] qsize 0 2021-04-08 14:53:36,534 DEBUG awx.main.dispatch task c4c457ca-db7c-4009-930b-37a7ea9e22d5 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:53:36,537 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:53:36,543 DEBUG awx.main.dispatch task 87a053df-7697-424b-88f1-67f29706e8f0 is finished 2021-04-08 14:53:36,552 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:53:36,581 DEBUG awx.main.dispatch task c4c457ca-db7c-4009-930b-37a7ea9e22d5 is finished 2021-04-08 14:53:56,521 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(35f3870b-1295-4818-a91d-fc2487050bc4, queue=awx_private_queue) [2021-04-08 14:53:56,521: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(35f3870b-1295-4818-a91d-fc2487050bc4, queue=awx_private_queue) 2021-04-08 14:53:56,544 DEBUG awx.main.dispatch delivered 35f3870b-1295-4818-a91d-fc2487050bc4 to worker[176] qsize 0 2021-04-08 14:53:56,551 DEBUG awx.main.dispatch task 35f3870b-1295-4818-a91d-fc2487050bc4 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:53:56,555 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:53:56,578 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:53:56,607 DEBUG awx.main.dispatch task 35f3870b-1295-4818-a91d-fc2487050bc4 is finished RESULT 2 OKREADY 2021-04-08 14:54:06,494 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4a32b64f-fafa-4300-b5fb-88094f1f0506, queue=awx_private_queue) [2021-04-08 14:54:06,494: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4a32b64f-fafa-4300-b5fb-88094f1f0506, queue=awx_private_queue) 2021-04-08 14:54:06,510 DEBUG awx.main.dispatch delivered 4a32b64f-fafa-4300-b5fb-88094f1f0506 to worker[178] qsize 0 2021-04-08 14:54:06,514 DEBUG awx.main.dispatch task 4a32b64f-fafa-4300-b5fb-88094f1f0506 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:54:06,707 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:54:06,730 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:53:36.526310+00:00 2021-04-08 14:54:06,805 DEBUG awx.main.dispatch task 4a32b64f-fafa-4300-b5fb-88094f1f0506 is finished 2021-04-08 14:54:16,518 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5e7a089b-888a-4e85-9d5a-80be56777e6a, queue=awx_private_queue) [2021-04-08 14:54:16,518: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5e7a089b-888a-4e85-9d5a-80be56777e6a, queue=awx_private_queue) 2021-04-08 14:54:16,535 DEBUG awx.main.dispatch delivered 5e7a089b-888a-4e85-9d5a-80be56777e6a to worker[178] qsize 0 2021-04-08 14:54:16,537 DEBUG awx.main.dispatch task 5e7a089b-888a-4e85-9d5a-80be56777e6a starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:54:16,539 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:54:16,558 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:54:16,601 DEBUG awx.main.dispatch task 5e7a089b-888a-4e85-9d5a-80be56777e6a is finished 2021-04-08 14:54:35,995 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(57d0cead-b641-4dab-be3c-8bbc013abb16, queue=dev-awx-0) [2021-04-08 14:54:35,995: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(57d0cead-b641-4dab-be3c-8bbc013abb16, queue=dev-awx-0) 2021-04-08 14:54:36,051 DEBUG awx.main.dispatch delivered 57d0cead-b641-4dab-be3c-8bbc013abb16 to worker[178] qsize 0 2021-04-08 14:54:36,053 DEBUG awx.main.dispatch task 57d0cead-b641-4dab-be3c-8bbc013abb16 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:54:36,054 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:54:36,132 DEBUG awx.main.dispatch task 57d0cead-b641-4dab-be3c-8bbc013abb16 is finished 2021-04-08 14:54:36,243 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(61480a5f-62aa-4b40-ac5e-958440ee50b8, queue=dev-awx-0) [2021-04-08 14:54:36,243: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(61480a5f-62aa-4b40-ac5e-958440ee50b8, queue=dev-awx-0) 2021-04-08 14:54:36,458 DEBUG awx.main.dispatch delivered 61480a5f-62aa-4b40-ac5e-958440ee50b8 to worker[177] qsize 0 2021-04-08 14:54:36,462 DEBUG awx.main.dispatch task 61480a5f-62aa-4b40-ac5e-958440ee50b8 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:54:36,480 DEBUG awx.main.dispatch task 61480a5f-62aa-4b40-ac5e-958440ee50b8 is finished 2021-04-08 14:54:36,495 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(985933eb-9756-4858-b1e3-3b2017307299, queue=awx_private_queue) [2021-04-08 14:54:36,495: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(985933eb-9756-4858-b1e3-3b2017307299, queue=awx_private_queue) 2021-04-08 14:54:36,532 DEBUG awx.main.dispatch delivered 985933eb-9756-4858-b1e3-3b2017307299 to worker[180] qsize 0 2021-04-08 14:54:36,534 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d322b549-e13b-48fd-bec5-2c386c4037e4, queue=awx_private_queue) [2021-04-08 14:54:36,534: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d322b549-e13b-48fd-bec5-2c386c4037e4, queue=awx_private_queue) 2021-04-08 14:54:36,536 DEBUG awx.main.dispatch task 985933eb-9756-4858-b1e3-3b2017307299 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:54:36,551 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:54:36,558 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:54:06.708073+00:00 2021-04-08 14:54:36,564 DEBUG awx.main.dispatch delivered d322b549-e13b-48fd-bec5-2c386c4037e4 to worker[177] qsize 0 2021-04-08 14:54:36,568 DEBUG awx.main.dispatch task d322b549-e13b-48fd-bec5-2c386c4037e4 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:54:36,570 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:54:36,577 DEBUG awx.main.dispatch task 985933eb-9756-4858-b1e3-3b2017307299 is finished 2021-04-08 14:54:36,586 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:54:36,634 DEBUG awx.main.dispatch task d322b549-e13b-48fd-bec5-2c386c4037e4 is finished 2021-04-08 14:54:56,534 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b20b2b2d-4843-40e6-ae42-19e42557e350, queue=awx_private_queue) [2021-04-08 14:54:56,534: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b20b2b2d-4843-40e6-ae42-19e42557e350, queue=awx_private_queue) 2021-04-08 14:54:56,550 DEBUG awx.main.dispatch delivered b20b2b2d-4843-40e6-ae42-19e42557e350 to worker[180] qsize 0 2021-04-08 14:54:56,553 DEBUG awx.main.dispatch task b20b2b2d-4843-40e6-ae42-19e42557e350 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:54:56,554 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:54:56,574 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:54:56,602 DEBUG awx.main.dispatch task b20b2b2d-4843-40e6-ae42-19e42557e350 is finished RESULT 2 OKREADY 2021-04-08 14:55:06,494 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(288ca372-7800-47a9-8148-02d168689402, queue=awx_private_queue) [2021-04-08 14:55:06,494: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(288ca372-7800-47a9-8148-02d168689402, queue=awx_private_queue) 2021-04-08 14:55:06,511 DEBUG awx.main.dispatch delivered 288ca372-7800-47a9-8148-02d168689402 to worker[180] qsize 0 2021-04-08 14:55:06,513 DEBUG awx.main.dispatch task 288ca372-7800-47a9-8148-02d168689402 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:55:06,646 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:55:06,707 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:54:36.551796+00:00 2021-04-08 14:55:06,767 DEBUG awx.main.dispatch task 288ca372-7800-47a9-8148-02d168689402 is finished 2021-04-08 14:55:16,533 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a79f8a61-06bb-424c-80f0-076dd8043a7a, queue=awx_private_queue) [2021-04-08 14:55:16,533: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a79f8a61-06bb-424c-80f0-076dd8043a7a, queue=awx_private_queue) 2021-04-08 14:55:16,549 DEBUG awx.main.dispatch delivered a79f8a61-06bb-424c-80f0-076dd8043a7a to worker[176] qsize 0 2021-04-08 14:55:16,551 DEBUG awx.main.dispatch task a79f8a61-06bb-424c-80f0-076dd8043a7a starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:55:16,554 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:55:16,570 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:55:16,596 DEBUG awx.main.dispatch task a79f8a61-06bb-424c-80f0-076dd8043a7a is finished 2021-04-08 14:55:36,001 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(1441492d-21e4-4324-80f5-042139709541, queue=dev-awx-0) [2021-04-08 14:55:36,001: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(1441492d-21e4-4324-80f5-042139709541, queue=dev-awx-0) 2021-04-08 14:55:36,041 DEBUG awx.main.dispatch delivered 1441492d-21e4-4324-80f5-042139709541 to worker[177] qsize 0 2021-04-08 14:55:36,043 DEBUG awx.main.dispatch task 1441492d-21e4-4324-80f5-042139709541 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:55:36,045 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:55:36,164 DEBUG awx.main.dispatch task 1441492d-21e4-4324-80f5-042139709541 is finished 2021-04-08 14:55:36,242 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(08e6f986-df40-46b7-81da-349f4a8f5a77, queue=dev-awx-0) [2021-04-08 14:55:36,242: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(08e6f986-df40-46b7-81da-349f4a8f5a77, queue=dev-awx-0) 2021-04-08 14:55:36,256 DEBUG awx.main.dispatch delivered 08e6f986-df40-46b7-81da-349f4a8f5a77 to worker[180] qsize 0 2021-04-08 14:55:36,258 DEBUG awx.main.dispatch task 08e6f986-df40-46b7-81da-349f4a8f5a77 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:55:36,270 DEBUG awx.main.dispatch task 08e6f986-df40-46b7-81da-349f4a8f5a77 is finished 2021-04-08 14:55:36,494 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(66e40ffd-6b25-4ba1-98e4-89f3632696c9, queue=awx_private_queue) [2021-04-08 14:55:36,494: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(66e40ffd-6b25-4ba1-98e4-89f3632696c9, queue=awx_private_queue) 2021-04-08 14:55:36,696 DEBUG awx.main.dispatch delivered 66e40ffd-6b25-4ba1-98e4-89f3632696c9 to worker[177] qsize 0 2021-04-08 14:55:36,698 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(27bb0689-4f34-4d3d-b19e-0fe1f6b22176, queue=awx_private_queue) [2021-04-08 14:55:36,698: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(27bb0689-4f34-4d3d-b19e-0fe1f6b22176, queue=awx_private_queue) 2021-04-08 14:55:36,699 DEBUG awx.main.dispatch task 66e40ffd-6b25-4ba1-98e4-89f3632696c9 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:55:36,709 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:55:36,711 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:55:06.646519+00:00 2021-04-08 14:55:36,711 DEBUG awx.main.dispatch delivered 27bb0689-4f34-4d3d-b19e-0fe1f6b22176 to worker[176] qsize 0 2021-04-08 14:55:36,713 DEBUG awx.main.dispatch task 27bb0689-4f34-4d3d-b19e-0fe1f6b22176 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:55:36,714 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:55:36,725 DEBUG awx.main.dispatch task 66e40ffd-6b25-4ba1-98e4-89f3632696c9 is finished 2021-04-08 14:55:36,730 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:55:36,761 DEBUG awx.main.dispatch task 27bb0689-4f34-4d3d-b19e-0fe1f6b22176 is finished 2021-04-08 14:55:56,720 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ee5df626-f494-4778-8d86-f3b35dc26a08, queue=awx_private_queue) [2021-04-08 14:55:56,720: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ee5df626-f494-4778-8d86-f3b35dc26a08, queue=awx_private_queue) 2021-04-08 14:55:56,761 DEBUG awx.main.dispatch delivered ee5df626-f494-4778-8d86-f3b35dc26a08 to worker[176] qsize 0 2021-04-08 14:55:56,772 DEBUG awx.main.dispatch task ee5df626-f494-4778-8d86-f3b35dc26a08 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:55:56,778 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:55:56,905 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:55:56,943 DEBUG awx.main.dispatch task ee5df626-f494-4778-8d86-f3b35dc26a08 is finished RESULT 2 OKREADY 2021-04-08 14:56:06,495 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e0cac29d-9550-437b-a467-9fe6496da75e, queue=awx_private_queue) [2021-04-08 14:56:06,495: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e0cac29d-9550-437b-a467-9fe6496da75e, queue=awx_private_queue) 2021-04-08 14:56:06,515 DEBUG awx.main.dispatch delivered e0cac29d-9550-437b-a467-9fe6496da75e to worker[180] qsize 0 2021-04-08 14:56:06,520 DEBUG awx.main.dispatch task e0cac29d-9550-437b-a467-9fe6496da75e starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:56:06,602 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:56:06,611 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:55:36.709178+00:00 2021-04-08 14:56:06,790 DEBUG awx.main.dispatch task e0cac29d-9550-437b-a467-9fe6496da75e is finished 2021-04-08 14:56:16,713 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ec8fbbae-45dd-4d02-b209-bde6e06ca065, queue=awx_private_queue) [2021-04-08 14:56:16,713: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ec8fbbae-45dd-4d02-b209-bde6e06ca065, queue=awx_private_queue) 2021-04-08 14:56:16,729 DEBUG awx.main.dispatch delivered ec8fbbae-45dd-4d02-b209-bde6e06ca065 to worker[180] qsize 0 2021-04-08 14:56:16,732 DEBUG awx.main.dispatch task ec8fbbae-45dd-4d02-b209-bde6e06ca065 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:56:16,733 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:56:16,756 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:56:16,798 DEBUG awx.main.dispatch task ec8fbbae-45dd-4d02-b209-bde6e06ca065 is finished 2021-04-08 14:56:36,010 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(9dcd5087-e08e-4e8d-8640-f0cc6ec54c89, queue=dev-awx-0) [2021-04-08 14:56:36,010: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(9dcd5087-e08e-4e8d-8640-f0cc6ec54c89, queue=dev-awx-0) 2021-04-08 14:56:36,117 DEBUG awx.main.dispatch delivered 9dcd5087-e08e-4e8d-8640-f0cc6ec54c89 to worker[180] qsize 0 2021-04-08 14:56:36,119 DEBUG awx.main.dispatch task 9dcd5087-e08e-4e8d-8640-f0cc6ec54c89 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:56:36,121 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:56:36,242 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(cb9d7a2a-b3ae-4e4d-817c-f353e3471d9d, queue=dev-awx-0) [2021-04-08 14:56:36,242: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(cb9d7a2a-b3ae-4e4d-817c-f353e3471d9d, queue=dev-awx-0) 2021-04-08 14:56:36,257 DEBUG awx.main.dispatch delivered cb9d7a2a-b3ae-4e4d-817c-f353e3471d9d to worker[176] qsize 0 2021-04-08 14:56:36,394 DEBUG awx.main.dispatch task 9dcd5087-e08e-4e8d-8640-f0cc6ec54c89 is finished 2021-04-08 14:56:36,260 DEBUG awx.main.dispatch task cb9d7a2a-b3ae-4e4d-817c-f353e3471d9d starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:56:36,489 DEBUG awx.main.dispatch task cb9d7a2a-b3ae-4e4d-817c-f353e3471d9d is finished 2021-04-08 14:56:36,494 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5350a0ef-ee28-474c-81b3-e8d63f7d87ad, queue=awx_private_queue) [2021-04-08 14:56:36,494: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5350a0ef-ee28-474c-81b3-e8d63f7d87ad, queue=awx_private_queue) 2021-04-08 14:56:36,510 DEBUG awx.main.dispatch delivered 5350a0ef-ee28-474c-81b3-e8d63f7d87ad to worker[180] qsize 0 2021-04-08 14:56:36,512 DEBUG awx.main.dispatch task 5350a0ef-ee28-474c-81b3-e8d63f7d87ad starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:56:36,522 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:56:36,525 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:56:06.603071+00:00 2021-04-08 14:56:36,538 DEBUG awx.main.dispatch task 5350a0ef-ee28-474c-81b3-e8d63f7d87ad is finished 2021-04-08 14:56:36,713 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(471a4506-7daa-4991-bbcf-73da9d942564, queue=awx_private_queue) [2021-04-08 14:56:36,713: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(471a4506-7daa-4991-bbcf-73da9d942564, queue=awx_private_queue) 2021-04-08 14:56:36,729 DEBUG awx.main.dispatch delivered 471a4506-7daa-4991-bbcf-73da9d942564 to worker[180] qsize 0 2021-04-08 14:56:36,732 DEBUG awx.main.dispatch task 471a4506-7daa-4991-bbcf-73da9d942564 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:56:36,733 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:56:36,749 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:56:36,779 DEBUG awx.main.dispatch task 471a4506-7daa-4991-bbcf-73da9d942564 is finished 2021-04-08 14:56:56,714 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1759cc22-d6ab-49f9-8b57-a16d89c181fe, queue=awx_private_queue) [2021-04-08 14:56:56,714: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1759cc22-d6ab-49f9-8b57-a16d89c181fe, queue=awx_private_queue) 2021-04-08 14:56:56,728 DEBUG awx.main.dispatch delivered 1759cc22-d6ab-49f9-8b57-a16d89c181fe to worker[178] qsize 0 2021-04-08 14:56:56,733 DEBUG awx.main.dispatch task 1759cc22-d6ab-49f9-8b57-a16d89c181fe starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:56:56,739 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:56:56,782 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:56:56,842 DEBUG awx.main.dispatch task 1759cc22-d6ab-49f9-8b57-a16d89c181fe is finished RESULT 2 OKREADY 2021-04-08 14:57:06,495 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(efce27bb-906b-49cf-96f0-36e38c14c26d, queue=awx_private_queue) [2021-04-08 14:57:06,495: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(efce27bb-906b-49cf-96f0-36e38c14c26d, queue=awx_private_queue) 2021-04-08 14:57:06,511 DEBUG awx.main.dispatch delivered efce27bb-906b-49cf-96f0-36e38c14c26d to worker[176] qsize 0 2021-04-08 14:57:06,514 DEBUG awx.main.dispatch task efce27bb-906b-49cf-96f0-36e38c14c26d starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:57:06,525 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:57:06,555 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:56:36.522944+00:00 2021-04-08 14:57:06,659 DEBUG awx.main.dispatch task efce27bb-906b-49cf-96f0-36e38c14c26d is finished 2021-04-08 14:57:16,714 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(57c9529c-db7a-45a8-a20c-21b09ad3534e, queue=awx_private_queue) [2021-04-08 14:57:16,714: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(57c9529c-db7a-45a8-a20c-21b09ad3534e, queue=awx_private_queue) 2021-04-08 14:57:16,728 DEBUG awx.main.dispatch delivered 57c9529c-db7a-45a8-a20c-21b09ad3534e to worker[180] qsize 0 2021-04-08 14:57:16,730 DEBUG awx.main.dispatch task 57c9529c-db7a-45a8-a20c-21b09ad3534e starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:57:16,733 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:57:16,749 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:57:16,781 DEBUG awx.main.dispatch task 57c9529c-db7a-45a8-a20c-21b09ad3534e is finished 2021-04-08 14:57:35,881 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(4ccf10cc-2215-4d92-a821-3432b0a24a7f, queue=dev-awx-0) [2021-04-08 14:57:35,881: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(4ccf10cc-2215-4d92-a821-3432b0a24a7f, queue=dev-awx-0) 2021-04-08 14:57:35,899 DEBUG awx.main.dispatch delivered 4ccf10cc-2215-4d92-a821-3432b0a24a7f to worker[180] qsize 0 2021-04-08 14:57:35,901 DEBUG awx.main.dispatch task 4ccf10cc-2215-4d92-a821-3432b0a24a7f starting awx.main.tasks.awx_isolated_heartbeat(*[]) 2021-04-08 14:57:35,903 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks. 2021-04-08 14:57:35,931 DEBUG awx.main.dispatch task 4ccf10cc-2215-4d92-a821-3432b0a24a7f is finished 2021-04-08 14:57:36,009 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b66acdda-b000-4d76-bc13-17bf372edfec, queue=dev-awx-0) [2021-04-08 14:57:36,009: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b66acdda-b000-4d76-bc13-17bf372edfec, queue=dev-awx-0) 2021-04-08 14:57:36,037 DEBUG awx.main.dispatch delivered b66acdda-b000-4d76-bc13-17bf372edfec to worker[177] qsize 0 2021-04-08 14:57:36,242 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(56e75d98-a371-4698-a8cb-57d98ef824fc, queue=dev-awx-0) [2021-04-08 14:57:36,242: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(56e75d98-a371-4698-a8cb-57d98ef824fc, queue=dev-awx-0) 2021-04-08 14:57:36,048 DEBUG awx.main.dispatch task b66acdda-b000-4d76-bc13-17bf372edfec starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:57:36,256 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:57:36,259 DEBUG awx.main.dispatch delivered 56e75d98-a371-4698-a8cb-57d98ef824fc to worker[178] qsize 0 2021-04-08 14:57:36,261 DEBUG awx.main.dispatch task 56e75d98-a371-4698-a8cb-57d98ef824fc starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:57:36,277 DEBUG awx.main.dispatch task 56e75d98-a371-4698-a8cb-57d98ef824fc is finished 2021-04-08 14:57:36,337 DEBUG awx.main.dispatch task b66acdda-b000-4d76-bc13-17bf372edfec is finished 2021-04-08 14:57:36,495 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dcf0b6c2-7a63-4638-8afa-4cd3fe638c2a, queue=awx_private_queue) [2021-04-08 14:57:36,495: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dcf0b6c2-7a63-4638-8afa-4cd3fe638c2a, queue=awx_private_queue) 2021-04-08 14:57:36,512 DEBUG awx.main.dispatch delivered dcf0b6c2-7a63-4638-8afa-4cd3fe638c2a to worker[177] qsize 0 2021-04-08 14:57:36,514 DEBUG awx.main.dispatch task dcf0b6c2-7a63-4638-8afa-4cd3fe638c2a starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:57:36,525 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:57:36,528 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:57:06.526251+00:00 2021-04-08 14:57:36,543 DEBUG awx.main.dispatch task dcf0b6c2-7a63-4638-8afa-4cd3fe638c2a is finished 2021-04-08 14:57:36,713 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e779b4f5-72c2-436f-a4e7-07684bd64515, queue=awx_private_queue) [2021-04-08 14:57:36,713: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e779b4f5-72c2-436f-a4e7-07684bd64515, queue=awx_private_queue) 2021-04-08 14:57:36,728 DEBUG awx.main.dispatch delivered e779b4f5-72c2-436f-a4e7-07684bd64515 to worker[178] qsize 0 2021-04-08 14:57:36,730 DEBUG awx.main.dispatch task e779b4f5-72c2-436f-a4e7-07684bd64515 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:57:36,732 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:57:36,748 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:57:36,785 DEBUG awx.main.dispatch task e779b4f5-72c2-436f-a4e7-07684bd64515 is finished 2021-04-08 14:57:56,716 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(066c8b59-26bf-4ca4-85d9-f3e981fdf457, queue=awx_private_queue) [2021-04-08 14:57:56,716: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(066c8b59-26bf-4ca4-85d9-f3e981fdf457, queue=awx_private_queue) 2021-04-08 14:57:56,739 DEBUG awx.main.dispatch delivered 066c8b59-26bf-4ca4-85d9-f3e981fdf457 to worker[176] qsize 0 2021-04-08 14:57:56,747 DEBUG awx.main.dispatch task 066c8b59-26bf-4ca4-85d9-f3e981fdf457 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:57:56,750 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:57:56,785 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:57:56,816 DEBUG awx.main.dispatch task 066c8b59-26bf-4ca4-85d9-f3e981fdf457 is finished RESULT 2 OKREADY 2021-04-08 14:58:06,497 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ce751543-ef74-4018-b426-76ee79361cf0, queue=awx_private_queue) [2021-04-08 14:58:06,497: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ce751543-ef74-4018-b426-76ee79361cf0, queue=awx_private_queue) 2021-04-08 14:58:06,524 DEBUG awx.main.dispatch delivered ce751543-ef74-4018-b426-76ee79361cf0 to worker[177] qsize 0 2021-04-08 14:58:06,528 DEBUG awx.main.dispatch task ce751543-ef74-4018-b426-76ee79361cf0 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:58:06,585 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:58:06,590 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:57:36.525653+00:00 2021-04-08 14:58:06,632 DEBUG awx.main.dispatch task ce751543-ef74-4018-b426-76ee79361cf0 is finished 2021-04-08 14:58:16,715 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(79e3dec3-7803-471d-bf8b-7fa671ecc9ca, queue=awx_private_queue) [2021-04-08 14:58:16,715: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(79e3dec3-7803-471d-bf8b-7fa671ecc9ca, queue=awx_private_queue) 2021-04-08 14:58:16,736 DEBUG awx.main.dispatch delivered 79e3dec3-7803-471d-bf8b-7fa671ecc9ca to worker[180] qsize 0 2021-04-08 14:58:16,740 DEBUG awx.main.dispatch task 79e3dec3-7803-471d-bf8b-7fa671ecc9ca starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:58:16,747 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:58:16,778 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:58:16,813 DEBUG awx.main.dispatch task 79e3dec3-7803-471d-bf8b-7fa671ecc9ca is finished 2021-04-08 14:58:36,009 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(bda5db92-38d2-451c-9ea8-c4b7d5c834e2, queue=dev-awx-0) [2021-04-08 14:58:36,009: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(bda5db92-38d2-451c-9ea8-c4b7d5c834e2, queue=dev-awx-0) 2021-04-08 14:58:36,048 DEBUG awx.main.dispatch delivered bda5db92-38d2-451c-9ea8-c4b7d5c834e2 to worker[177] qsize 0 2021-04-08 14:58:36,050 DEBUG awx.main.dispatch task bda5db92-38d2-451c-9ea8-c4b7d5c834e2 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:58:36,052 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:58:36,147 DEBUG awx.main.dispatch task bda5db92-38d2-451c-9ea8-c4b7d5c834e2 is finished 2021-04-08 14:58:36,242 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ab92f9e5-cbb8-4759-b4c5-a22c0f8308f7, queue=dev-awx-0) [2021-04-08 14:58:36,242: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ab92f9e5-cbb8-4759-b4c5-a22c0f8308f7, queue=dev-awx-0) 2021-04-08 14:58:36,258 DEBUG awx.main.dispatch delivered ab92f9e5-cbb8-4759-b4c5-a22c0f8308f7 to worker[176] qsize 0 2021-04-08 14:58:36,261 DEBUG awx.main.dispatch task ab92f9e5-cbb8-4759-b4c5-a22c0f8308f7 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:58:36,277 DEBUG awx.main.dispatch task ab92f9e5-cbb8-4759-b4c5-a22c0f8308f7 is finished 2021-04-08 14:58:36,495 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(290bdb5c-080c-4b03-a875-cd0a9f47ea37, queue=awx_private_queue) [2021-04-08 14:58:36,495: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(290bdb5c-080c-4b03-a875-cd0a9f47ea37, queue=awx_private_queue) 2021-04-08 14:58:36,709 DEBUG awx.main.dispatch delivered 290bdb5c-080c-4b03-a875-cd0a9f47ea37 to worker[180] qsize 0 2021-04-08 14:58:36,711 DEBUG awx.main.dispatch task 290bdb5c-080c-4b03-a875-cd0a9f47ea37 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:58:36,715 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(af1f63ae-d559-40f1-a8a4-dbb2194e1679, queue=awx_private_queue) [2021-04-08 14:58:36,715: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(af1f63ae-d559-40f1-a8a4-dbb2194e1679, queue=awx_private_queue) 2021-04-08 14:58:36,731 DEBUG awx.main.dispatch delivered af1f63ae-d559-40f1-a8a4-dbb2194e1679 to worker[178] qsize 0 2021-04-08 14:58:36,734 DEBUG awx.main.dispatch task af1f63ae-d559-40f1-a8a4-dbb2194e1679 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:58:36,738 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:58:36,953 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:58:37,050 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:58:37,054 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:58:06.585534+00:00 2021-04-08 14:58:37,086 DEBUG awx.main.dispatch task 290bdb5c-080c-4b03-a875-cd0a9f47ea37 is finished 2021-04-08 14:58:37,303 DEBUG awx.main.dispatch task af1f63ae-d559-40f1-a8a4-dbb2194e1679 is finished 2021-04-08 14:58:56,727 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(96589647-3160-4566-b961-5b15826671ff, queue=awx_private_queue) [2021-04-08 14:58:56,727: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(96589647-3160-4566-b961-5b15826671ff, queue=awx_private_queue) 2021-04-08 14:58:56,742 DEBUG awx.main.dispatch delivered 96589647-3160-4566-b961-5b15826671ff to worker[178] qsize 0 2021-04-08 14:58:56,744 DEBUG awx.main.dispatch task 96589647-3160-4566-b961-5b15826671ff starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:58:56,746 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:58:56,764 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:58:56,794 DEBUG awx.main.dispatch task 96589647-3160-4566-b961-5b15826671ff is finished RESULT 2 OKREADY 2021-04-08 14:59:06,496 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b18a0625-d721-43d4-9fc6-b8cba08dfcd3, queue=awx_private_queue) [2021-04-08 14:59:06,496: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b18a0625-d721-43d4-9fc6-b8cba08dfcd3, queue=awx_private_queue) 2021-04-08 14:59:06,511 DEBUG awx.main.dispatch delivered b18a0625-d721-43d4-9fc6-b8cba08dfcd3 to worker[177] qsize 0 2021-04-08 14:59:06,513 DEBUG awx.main.dispatch task b18a0625-d721-43d4-9fc6-b8cba08dfcd3 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:59:08,627 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:59:08,749 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:58:37.050849+00:00 2021-04-08 14:59:08,917 DEBUG awx.main.dispatch task b18a0625-d721-43d4-9fc6-b8cba08dfcd3 is finished 2021-04-08 14:59:16,728 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a0c89212-0d60-4758-ae8f-ab481ce9cf45, queue=awx_private_queue) [2021-04-08 14:59:16,728: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a0c89212-0d60-4758-ae8f-ab481ce9cf45, queue=awx_private_queue) 2021-04-08 14:59:16,747 DEBUG awx.main.dispatch delivered a0c89212-0d60-4758-ae8f-ab481ce9cf45 to worker[177] qsize 0 2021-04-08 14:59:16,751 DEBUG awx.main.dispatch task a0c89212-0d60-4758-ae8f-ab481ce9cf45 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:59:16,753 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:59:16,842 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:59:16,882 DEBUG awx.main.dispatch task a0c89212-0d60-4758-ae8f-ab481ce9cf45 is finished 2021-04-08 14:59:36,011 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c7ac5f4d-fff4-4030-a488-54701866e701, queue=dev-awx-0) [2021-04-08 14:59:36,011: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c7ac5f4d-fff4-4030-a488-54701866e701, queue=dev-awx-0) 2021-04-08 14:59:36,075 DEBUG awx.main.dispatch delivered c7ac5f4d-fff4-4030-a488-54701866e701 to worker[180] qsize 0 2021-04-08 14:59:36,078 DEBUG awx.main.dispatch task c7ac5f4d-fff4-4030-a488-54701866e701 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 14:59:36,080 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 14:59:36,203 DEBUG awx.main.dispatch task c7ac5f4d-fff4-4030-a488-54701866e701 is finished 2021-04-08 14:59:36,242 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(115b37a6-95e0-48e5-a134-c9e7a17cf474, queue=dev-awx-0) [2021-04-08 14:59:36,242: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(115b37a6-95e0-48e5-a134-c9e7a17cf474, queue=dev-awx-0) 2021-04-08 14:59:36,259 DEBUG awx.main.dispatch delivered 115b37a6-95e0-48e5-a134-c9e7a17cf474 to worker[176] qsize 0 2021-04-08 14:59:36,262 DEBUG awx.main.dispatch task 115b37a6-95e0-48e5-a134-c9e7a17cf474 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 14:59:36,457 DEBUG awx.main.dispatch task 115b37a6-95e0-48e5-a134-c9e7a17cf474 is finished 2021-04-08 14:59:36,495 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bd437d74-8434-43dc-81e1-61182c1e74cf, queue=awx_private_queue) [2021-04-08 14:59:36,495: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bd437d74-8434-43dc-81e1-61182c1e74cf, queue=awx_private_queue) 2021-04-08 14:59:36,523 DEBUG awx.main.dispatch delivered bd437d74-8434-43dc-81e1-61182c1e74cf to worker[178] qsize 0 2021-04-08 14:59:36,525 DEBUG awx.main.dispatch task bd437d74-8434-43dc-81e1-61182c1e74cf starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 14:59:36,535 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 14:59:36,538 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:59:08.628110+00:00 2021-04-08 14:59:36,556 DEBUG awx.main.dispatch task bd437d74-8434-43dc-81e1-61182c1e74cf is finished 2021-04-08 14:59:36,728 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bae8686b-43f3-4e16-b240-66663047e1c4, queue=awx_private_queue) [2021-04-08 14:59:36,728: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bae8686b-43f3-4e16-b240-66663047e1c4, queue=awx_private_queue) 2021-04-08 14:59:36,758 DEBUG awx.main.dispatch delivered bae8686b-43f3-4e16-b240-66663047e1c4 to worker[180] qsize 0 2021-04-08 14:59:36,760 DEBUG awx.main.dispatch task bae8686b-43f3-4e16-b240-66663047e1c4 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:59:36,761 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:59:36,778 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:59:36,811 DEBUG awx.main.dispatch task bae8686b-43f3-4e16-b240-66663047e1c4 is finished 2021-04-08 14:59:56,737 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ea64843a-1957-4ed9-9fbf-cdf742b4094f, queue=awx_private_queue) [2021-04-08 14:59:56,737: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ea64843a-1957-4ed9-9fbf-cdf742b4094f, queue=awx_private_queue) 2021-04-08 14:59:56,763 DEBUG awx.main.dispatch delivered ea64843a-1957-4ed9-9fbf-cdf742b4094f to worker[178] qsize 0 2021-04-08 14:59:56,770 DEBUG awx.main.dispatch task ea64843a-1957-4ed9-9fbf-cdf742b4094f starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 14:59:56,773 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 14:59:56,908 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 14:59:56,942 DEBUG awx.main.dispatch task ea64843a-1957-4ed9-9fbf-cdf742b4094f is finished RESULT 2 OKREADY 2021-04-08 15:00:06,498 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(406f01c8-7063-47cc-acf4-93492ad0d7b1, queue=awx_private_queue) [2021-04-08 15:00:06,498: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(406f01c8-7063-47cc-acf4-93492ad0d7b1, queue=awx_private_queue) 2021-04-08 15:00:06,518 DEBUG awx.main.dispatch delivered 406f01c8-7063-47cc-acf4-93492ad0d7b1 to worker[177] qsize 0 2021-04-08 15:00:06,523 DEBUG awx.main.dispatch task 406f01c8-7063-47cc-acf4-93492ad0d7b1 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:00:07,566 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:00:08,022 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 14:59:36.535535+00:00 2021-04-08 15:00:08,153 DEBUG awx.main.dispatch task 406f01c8-7063-47cc-acf4-93492ad0d7b1 is finished 2021-04-08 15:00:16,735 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(64ef038c-a001-4b10-acea-bff158ba5f81, queue=awx_private_queue) [2021-04-08 15:00:16,735: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(64ef038c-a001-4b10-acea-bff158ba5f81, queue=awx_private_queue) 2021-04-08 15:00:16,759 DEBUG awx.main.dispatch delivered 64ef038c-a001-4b10-acea-bff158ba5f81 to worker[176] qsize 0 2021-04-08 15:00:16,763 DEBUG awx.main.dispatch task 64ef038c-a001-4b10-acea-bff158ba5f81 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:00:16,767 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:00:16,791 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:00:16,858 DEBUG awx.main.dispatch task 64ef038c-a001-4b10-acea-bff158ba5f81 is finished 2021-04-08 15:00:36,020 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d24716c2-2f6a-4cf3-a3b6-fc2d73137574, queue=dev-awx-0) [2021-04-08 15:00:36,020: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d24716c2-2f6a-4cf3-a3b6-fc2d73137574, queue=dev-awx-0) 2021-04-08 15:00:36,130 DEBUG awx.main.dispatch delivered d24716c2-2f6a-4cf3-a3b6-fc2d73137574 to worker[180] qsize 0 2021-04-08 15:00:36,134 DEBUG awx.main.dispatch task d24716c2-2f6a-4cf3-a3b6-fc2d73137574 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 15:00:36,138 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 15:00:36,230 DEBUG awx.main.dispatch task d24716c2-2f6a-4cf3-a3b6-fc2d73137574 is finished 2021-04-08 15:00:36,242 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(9e54333a-6141-48d5-bf46-5334c9d1aff4, queue=dev-awx-0) [2021-04-08 15:00:36,242: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(9e54333a-6141-48d5-bf46-5334c9d1aff4, queue=dev-awx-0) 2021-04-08 15:00:36,257 DEBUG awx.main.dispatch delivered 9e54333a-6141-48d5-bf46-5334c9d1aff4 to worker[176] qsize 0 2021-04-08 15:00:36,259 DEBUG awx.main.dispatch task 9e54333a-6141-48d5-bf46-5334c9d1aff4 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:00:36,273 DEBUG awx.main.dispatch task 9e54333a-6141-48d5-bf46-5334c9d1aff4 is finished 2021-04-08 15:00:36,497 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(575247ea-d020-414e-991a-06ee6f42d3a1, queue=awx_private_queue) [2021-04-08 15:00:36,497: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(575247ea-d020-414e-991a-06ee6f42d3a1, queue=awx_private_queue) 2021-04-08 15:00:36,513 DEBUG awx.main.dispatch delivered 575247ea-d020-414e-991a-06ee6f42d3a1 to worker[176] qsize 0 2021-04-08 15:00:36,515 DEBUG awx.main.dispatch task 575247ea-d020-414e-991a-06ee6f42d3a1 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:00:36,654 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:00:36,659 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:00:07.566304+00:00 2021-04-08 15:00:36,672 DEBUG awx.main.dispatch task 575247ea-d020-414e-991a-06ee6f42d3a1 is finished 2021-04-08 15:00:36,735 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c9c69c7f-ec8e-402f-bf8c-1f7f3f80f2a4, queue=awx_private_queue) [2021-04-08 15:00:36,735: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c9c69c7f-ec8e-402f-bf8c-1f7f3f80f2a4, queue=awx_private_queue) 2021-04-08 15:00:36,758 DEBUG awx.main.dispatch delivered c9c69c7f-ec8e-402f-bf8c-1f7f3f80f2a4 to worker[176] qsize 0 2021-04-08 15:00:36,761 DEBUG awx.main.dispatch task c9c69c7f-ec8e-402f-bf8c-1f7f3f80f2a4 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:00:36,762 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:00:36,779 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:00:36,811 DEBUG awx.main.dispatch task c9c69c7f-ec8e-402f-bf8c-1f7f3f80f2a4 is finished 2021-04-08 15:00:56,735 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0eeee18c-49ce-4c24-9f6f-81d5c0b20ff3, queue=awx_private_queue) [2021-04-08 15:00:56,735: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0eeee18c-49ce-4c24-9f6f-81d5c0b20ff3, queue=awx_private_queue) 2021-04-08 15:00:56,753 DEBUG awx.main.dispatch delivered 0eeee18c-49ce-4c24-9f6f-81d5c0b20ff3 to worker[180] qsize 0 2021-04-08 15:00:56,755 DEBUG awx.main.dispatch task 0eeee18c-49ce-4c24-9f6f-81d5c0b20ff3 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:00:56,756 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:00:57,442 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:00:57,549 DEBUG awx.main.dispatch task 0eeee18c-49ce-4c24-9f6f-81d5c0b20ff3 is finished RESULT 2 OKREADY 2021-04-08 15:01:06,496 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3bd5bd19-5055-4d22-acef-0fccf1d7b2d5, queue=awx_private_queue) [2021-04-08 15:01:06,496: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3bd5bd19-5055-4d22-acef-0fccf1d7b2d5, queue=awx_private_queue) 2021-04-08 15:01:06,511 DEBUG awx.main.dispatch delivered 3bd5bd19-5055-4d22-acef-0fccf1d7b2d5 to worker[180] qsize 0 2021-04-08 15:01:06,514 DEBUG awx.main.dispatch task 3bd5bd19-5055-4d22-acef-0fccf1d7b2d5 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:01:06,692 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:01:06,696 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:00:36.654657+00:00 2021-04-08 15:01:06,835 DEBUG awx.main.dispatch task 3bd5bd19-5055-4d22-acef-0fccf1d7b2d5 is finished 2021-04-08 15:01:16,735 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d39f05ba-a9fc-45a7-9bbc-a2a5a9f9caba, queue=awx_private_queue) [2021-04-08 15:01:16,735: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d39f05ba-a9fc-45a7-9bbc-a2a5a9f9caba, queue=awx_private_queue) 2021-04-08 15:01:16,753 DEBUG awx.main.dispatch delivered d39f05ba-a9fc-45a7-9bbc-a2a5a9f9caba to worker[180] qsize 0 2021-04-08 15:01:16,757 DEBUG awx.main.dispatch task d39f05ba-a9fc-45a7-9bbc-a2a5a9f9caba starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:01:16,758 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:01:16,777 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:01:16,805 DEBUG awx.main.dispatch task d39f05ba-a9fc-45a7-9bbc-a2a5a9f9caba is finished 2021-04-08 15:01:36,027 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(1898285b-4909-4ee2-9155-7f9313a69e46, queue=dev-awx-0) [2021-04-08 15:01:36,027: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(1898285b-4909-4ee2-9155-7f9313a69e46, queue=dev-awx-0) 2021-04-08 15:01:36,122 DEBUG awx.main.dispatch delivered 1898285b-4909-4ee2-9155-7f9313a69e46 to worker[176] qsize 0 2021-04-08 15:01:36,124 DEBUG awx.main.dispatch task 1898285b-4909-4ee2-9155-7f9313a69e46 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 15:01:36,127 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 15:01:36,243 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(66633ea1-0f29-4593-985f-afdabdc557a1, queue=dev-awx-0) [2021-04-08 15:01:36,243: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(66633ea1-0f29-4593-985f-afdabdc557a1, queue=dev-awx-0) 2021-04-08 15:01:36,268 DEBUG awx.main.dispatch delivered 66633ea1-0f29-4593-985f-afdabdc557a1 to worker[177] qsize 0 2021-04-08 15:01:36,309 DEBUG awx.main.dispatch task 1898285b-4909-4ee2-9155-7f9313a69e46 is finished 2021-04-08 15:01:36,273 DEBUG awx.main.dispatch task 66633ea1-0f29-4593-985f-afdabdc557a1 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:01:36,496 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b3fb037e-5077-4ffd-94ea-53de8df45162, queue=awx_private_queue) [2021-04-08 15:01:36,496: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b3fb037e-5077-4ffd-94ea-53de8df45162, queue=awx_private_queue) 2021-04-08 15:01:36,501 DEBUG awx.main.dispatch task 66633ea1-0f29-4593-985f-afdabdc557a1 is finished 2021-04-08 15:01:36,510 DEBUG awx.main.dispatch delivered b3fb037e-5077-4ffd-94ea-53de8df45162 to worker[177] qsize 0 2021-04-08 15:01:36,512 DEBUG awx.main.dispatch task b3fb037e-5077-4ffd-94ea-53de8df45162 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:01:36,520 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:01:36,522 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:01:06.693006+00:00 2021-04-08 15:01:36,537 DEBUG awx.main.dispatch task b3fb037e-5077-4ffd-94ea-53de8df45162 is finished 2021-04-08 15:01:36,735 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bda4d6ce-243e-482e-a92c-3761ef8203aa, queue=awx_private_queue) [2021-04-08 15:01:36,735: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bda4d6ce-243e-482e-a92c-3761ef8203aa, queue=awx_private_queue) 2021-04-08 15:01:36,753 DEBUG awx.main.dispatch delivered bda4d6ce-243e-482e-a92c-3761ef8203aa to worker[177] qsize 0 2021-04-08 15:01:36,756 DEBUG awx.main.dispatch task bda4d6ce-243e-482e-a92c-3761ef8203aa starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:01:36,757 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:01:36,774 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:01:36,804 DEBUG awx.main.dispatch task bda4d6ce-243e-482e-a92c-3761ef8203aa is finished 2021-04-08 15:01:56,744 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0c0558d8-1e6c-4943-bcef-ac8773d019f7, queue=awx_private_queue) [2021-04-08 15:01:56,744: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0c0558d8-1e6c-4943-bcef-ac8773d019f7, queue=awx_private_queue) 2021-04-08 15:01:56,784 DEBUG awx.main.dispatch delivered 0c0558d8-1e6c-4943-bcef-ac8773d019f7 to worker[176] qsize 0 2021-04-08 15:01:56,785 DEBUG awx.main.dispatch task 0c0558d8-1e6c-4943-bcef-ac8773d019f7 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:01:56,787 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:01:57,098 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:01:57,147 DEBUG awx.main.dispatch task 0c0558d8-1e6c-4943-bcef-ac8773d019f7 is finished RESULT 2 OKREADY 2021-04-08 15:02:06,497 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0cef2110-1404-42f4-9169-cbe3cd404a0b, queue=awx_private_queue) [2021-04-08 15:02:06,497: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0cef2110-1404-42f4-9169-cbe3cd404a0b, queue=awx_private_queue) 2021-04-08 15:02:06,519 DEBUG awx.main.dispatch delivered 0cef2110-1404-42f4-9169-cbe3cd404a0b to worker[180] qsize 0 2021-04-08 15:02:06,522 DEBUG awx.main.dispatch task 0cef2110-1404-42f4-9169-cbe3cd404a0b starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:02:07,555 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:02:07,564 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:01:36.520249+00:00 2021-04-08 15:02:07,637 DEBUG awx.main.dispatch task 0cef2110-1404-42f4-9169-cbe3cd404a0b is finished 2021-04-08 15:02:16,756 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(aa4cb382-0855-48cf-970d-44723e8bb735, queue=awx_private_queue) [2021-04-08 15:02:16,756: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(aa4cb382-0855-48cf-970d-44723e8bb735, queue=awx_private_queue) 2021-04-08 15:02:16,787 DEBUG awx.main.dispatch delivered aa4cb382-0855-48cf-970d-44723e8bb735 to worker[177] qsize 0 2021-04-08 15:02:16,795 DEBUG awx.main.dispatch task aa4cb382-0855-48cf-970d-44723e8bb735 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:02:16,798 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:02:16,824 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:02:16,853 DEBUG awx.main.dispatch task aa4cb382-0855-48cf-970d-44723e8bb735 is finished 2021-04-08 15:02:36,036 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b65486e6-b035-456f-affe-4db119fa30e3, queue=dev-awx-0) [2021-04-08 15:02:36,036: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b65486e6-b035-456f-affe-4db119fa30e3, queue=dev-awx-0) 2021-04-08 15:02:36,118 DEBUG awx.main.dispatch delivered b65486e6-b035-456f-affe-4db119fa30e3 to worker[180] qsize 0 2021-04-08 15:02:36,120 DEBUG awx.main.dispatch task b65486e6-b035-456f-affe-4db119fa30e3 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 15:02:36,122 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 15:02:36,243 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2b799aa3-363d-47ab-9e7c-3c6380444d58, queue=dev-awx-0) [2021-04-08 15:02:36,243: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2b799aa3-363d-47ab-9e7c-3c6380444d58, queue=dev-awx-0) 2021-04-08 15:02:36,261 DEBUG awx.main.dispatch delivered 2b799aa3-363d-47ab-9e7c-3c6380444d58 to worker[178] qsize 0 2021-04-08 15:02:36,265 DEBUG awx.main.dispatch task 2b799aa3-363d-47ab-9e7c-3c6380444d58 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:02:36,275 DEBUG awx.main.dispatch task b65486e6-b035-456f-affe-4db119fa30e3 is finished 2021-04-08 15:02:36,288 DEBUG awx.main.dispatch task 2b799aa3-363d-47ab-9e7c-3c6380444d58 is finished 2021-04-08 15:02:36,496 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(593b1422-1bfd-470d-b0f9-b9563f54d84e, queue=awx_private_queue) [2021-04-08 15:02:36,496: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(593b1422-1bfd-470d-b0f9-b9563f54d84e, queue=awx_private_queue) 2021-04-08 15:02:36,918 DEBUG awx.main.dispatch delivered 593b1422-1bfd-470d-b0f9-b9563f54d84e to worker[176] qsize 0 2021-04-08 15:02:36,919 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0be36eb9-8c93-4501-8505-9380d5486f7c, queue=awx_private_queue) [2021-04-08 15:02:36,919: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0be36eb9-8c93-4501-8505-9380d5486f7c, queue=awx_private_queue) 2021-04-08 15:02:37,112 DEBUG awx.main.dispatch delivered 0be36eb9-8c93-4501-8505-9380d5486f7c to worker[177] qsize 0 2021-04-08 15:02:37,114 DEBUG awx.main.dispatch task 0be36eb9-8c93-4501-8505-9380d5486f7c starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:02:37,116 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:02:37,133 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:02:36,921 DEBUG awx.main.dispatch task 593b1422-1bfd-470d-b0f9-b9563f54d84e starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:02:37,138 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:02:37,140 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:02:07.556305+00:00 2021-04-08 15:02:37,157 DEBUG awx.main.dispatch task 593b1422-1bfd-470d-b0f9-b9563f54d84e is finished 2021-04-08 15:02:37,164 DEBUG awx.main.dispatch task 0be36eb9-8c93-4501-8505-9380d5486f7c is finished 2021-04-08 15:02:56,928 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4037993c-d9e3-45f1-9606-3702a6802f55, queue=awx_private_queue) [2021-04-08 15:02:56,928: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4037993c-d9e3-45f1-9606-3702a6802f55, queue=awx_private_queue) 2021-04-08 15:02:56,954 DEBUG awx.main.dispatch delivered 4037993c-d9e3-45f1-9606-3702a6802f55 to worker[176] qsize 0 2021-04-08 15:02:56,957 DEBUG awx.main.dispatch task 4037993c-d9e3-45f1-9606-3702a6802f55 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:02:56,958 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:02:57,278 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:02:57,312 DEBUG awx.main.dispatch task 4037993c-d9e3-45f1-9606-3702a6802f55 is finished RESULT 2 OKREADY 2021-04-08 15:03:06,496 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(641a305e-6d09-44c2-b9d3-c36cf947ebd3, queue=awx_private_queue) [2021-04-08 15:03:06,496: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(641a305e-6d09-44c2-b9d3-c36cf947ebd3, queue=awx_private_queue) 2021-04-08 15:03:06,518 DEBUG awx.main.dispatch delivered 641a305e-6d09-44c2-b9d3-c36cf947ebd3 to worker[176] qsize 0 2021-04-08 15:03:06,521 DEBUG awx.main.dispatch task 641a305e-6d09-44c2-b9d3-c36cf947ebd3 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:03:08,096 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:03:08,205 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:02:37.138582+00:00 2021-04-08 15:03:08,331 DEBUG awx.main.dispatch task 641a305e-6d09-44c2-b9d3-c36cf947ebd3 is finished 2021-04-08 15:03:16,927 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6ee39a71-ce54-4b80-bc77-d3adc0533a3f, queue=awx_private_queue) [2021-04-08 15:03:16,927: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6ee39a71-ce54-4b80-bc77-d3adc0533a3f, queue=awx_private_queue) 2021-04-08 15:03:16,946 DEBUG awx.main.dispatch delivered 6ee39a71-ce54-4b80-bc77-d3adc0533a3f to worker[180] qsize 0 2021-04-08 15:03:16,948 DEBUG awx.main.dispatch task 6ee39a71-ce54-4b80-bc77-d3adc0533a3f starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:03:16,950 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:03:17,020 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:03:17,062 DEBUG awx.main.dispatch task 6ee39a71-ce54-4b80-bc77-d3adc0533a3f is finished 2021-04-08 15:03:36,039 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5d066456-5162-4249-9f00-7fcc3abe6c28, queue=dev-awx-0) [2021-04-08 15:03:36,039: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5d066456-5162-4249-9f00-7fcc3abe6c28, queue=dev-awx-0) 2021-04-08 15:03:36,183 DEBUG awx.main.dispatch delivered 5d066456-5162-4249-9f00-7fcc3abe6c28 to worker[180] qsize 0 2021-04-08 15:03:36,185 DEBUG awx.main.dispatch task 5d066456-5162-4249-9f00-7fcc3abe6c28 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 15:03:36,187 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 15:03:36,243 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8be4bc57-45ab-42db-9be6-b62505ab68a1, queue=dev-awx-0) [2021-04-08 15:03:36,243: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8be4bc57-45ab-42db-9be6-b62505ab68a1, queue=dev-awx-0) 2021-04-08 15:03:36,261 DEBUG awx.main.dispatch delivered 8be4bc57-45ab-42db-9be6-b62505ab68a1 to worker[176] qsize 0 2021-04-08 15:03:36,263 DEBUG awx.main.dispatch task 8be4bc57-45ab-42db-9be6-b62505ab68a1 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:03:36,282 DEBUG awx.main.dispatch task 8be4bc57-45ab-42db-9be6-b62505ab68a1 is finished 2021-04-08 15:03:36,313 DEBUG awx.main.dispatch task 5d066456-5162-4249-9f00-7fcc3abe6c28 is finished 2021-04-08 15:03:36,496 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7d7a5cab-afb4-4197-b74f-a083f7f0825b, queue=awx_private_queue) [2021-04-08 15:03:36,496: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7d7a5cab-afb4-4197-b74f-a083f7f0825b, queue=awx_private_queue) 2021-04-08 15:03:36,526 DEBUG awx.main.dispatch delivered 7d7a5cab-afb4-4197-b74f-a083f7f0825b to worker[180] qsize 0 2021-04-08 15:03:36,528 DEBUG awx.main.dispatch task 7d7a5cab-afb4-4197-b74f-a083f7f0825b starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:03:36,536 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:03:36,538 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:03:08.097271+00:00 2021-04-08 15:03:36,564 DEBUG awx.main.dispatch task 7d7a5cab-afb4-4197-b74f-a083f7f0825b is finished 2021-04-08 15:03:36,927 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1ce87dad-2647-482a-9e9a-2d2553592215, queue=awx_private_queue) [2021-04-08 15:03:36,927: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1ce87dad-2647-482a-9e9a-2d2553592215, queue=awx_private_queue) 2021-04-08 15:03:37,127 DEBUG awx.main.dispatch delivered 1ce87dad-2647-482a-9e9a-2d2553592215 to worker[177] qsize 0 2021-04-08 15:03:37,129 DEBUG awx.main.dispatch task 1ce87dad-2647-482a-9e9a-2d2553592215 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:03:37,132 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:03:37,148 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:03:37,185 DEBUG awx.main.dispatch task 1ce87dad-2647-482a-9e9a-2d2553592215 is finished 2021-04-08 15:03:56,937 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7db44fb3-538e-4bfa-9939-832acade262f, queue=awx_private_queue) [2021-04-08 15:03:56,937: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7db44fb3-538e-4bfa-9939-832acade262f, queue=awx_private_queue) 2021-04-08 15:03:56,966 DEBUG awx.main.dispatch delivered 7db44fb3-538e-4bfa-9939-832acade262f to worker[180] qsize 0 2021-04-08 15:03:56,968 DEBUG awx.main.dispatch task 7db44fb3-538e-4bfa-9939-832acade262f starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:03:56,970 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:03:57,621 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:03:57,683 DEBUG awx.main.dispatch task 7db44fb3-538e-4bfa-9939-832acade262f is finished RESULT 2 OKREADY 2021-04-08 15:04:06,496 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(aa534deb-b169-4308-afa7-559c753038ab, queue=awx_private_queue) [2021-04-08 15:04:06,496: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(aa534deb-b169-4308-afa7-559c753038ab, queue=awx_private_queue) 2021-04-08 15:04:06,510 DEBUG awx.main.dispatch delivered aa534deb-b169-4308-afa7-559c753038ab to worker[177] qsize 0 2021-04-08 15:04:06,512 DEBUG awx.main.dispatch task aa534deb-b169-4308-afa7-559c753038ab starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:04:08,816 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:04:09,479 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:03:36.536720+00:00 2021-04-08 15:04:09,622 DEBUG awx.main.dispatch task aa534deb-b169-4308-afa7-559c753038ab is finished 2021-04-08 15:04:16,943 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(70ca69e4-b321-4419-85ad-16417f24423b, queue=awx_private_queue) [2021-04-08 15:04:16,943: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(70ca69e4-b321-4419-85ad-16417f24423b, queue=awx_private_queue) 2021-04-08 15:04:16,972 DEBUG awx.main.dispatch delivered 70ca69e4-b321-4419-85ad-16417f24423b to worker[178] qsize 0 2021-04-08 15:04:16,979 DEBUG awx.main.dispatch task 70ca69e4-b321-4419-85ad-16417f24423b starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:04:16,983 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:04:36,046 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(584706e3-76dc-41f3-a55e-a0b03ca55811, queue=dev-awx-0) [2021-04-08 15:04:36,046: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(584706e3-76dc-41f3-a55e-a0b03ca55811, queue=dev-awx-0) 2021-04-08 15:04:36,243 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(1c5dc528-607e-4a66-9e04-f1cdb35202fc, queue=dev-awx-0) [2021-04-08 15:04:36,243: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(1c5dc528-607e-4a66-9e04-f1cdb35202fc, queue=dev-awx-0) RESULT 2 OKREADY 2021-04-08 15:04:36,496 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5d511f85-a93b-410f-8358-1f47adbb61ee, queue=awx_private_queue) [2021-04-08 15:04:36,496: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5d511f85-a93b-410f-8358-1f47adbb61ee, queue=awx_private_queue) 2021-04-08 15:05:16,635 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:05:16,653 DEBUG awx.main.dispatch delivered 584706e3-76dc-41f3-a55e-a0b03ca55811 to worker[180] qsize 0 2021-04-08 15:05:16,656 DEBUG awx.main.dispatch delivered 1c5dc528-607e-4a66-9e04-f1cdb35202fc to worker[177] qsize 0 2021-04-08 15:05:16,659 DEBUG awx.main.dispatch delivered 5d511f85-a93b-410f-8358-1f47adbb61ee to worker[176] qsize 0 2021-04-08 15:05:16,822 DEBUG awx.main.dispatch task 70ca69e4-b321-4419-85ad-16417f24423b is finished 2021-04-08 15:05:16,659 DEBUG awx.main.dispatch task 1c5dc528-607e-4a66-9e04-f1cdb35202fc starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:05:16,920 DEBUG awx.main.dispatch task 1c5dc528-607e-4a66-9e04-f1cdb35202fc is finished 2021-04-08 15:05:16,663 DEBUG awx.main.dispatch task 5d511f85-a93b-410f-8358-1f47adbb61ee starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:05:16,962 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:05:16,642 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(267ca291-c49e-4555-a03b-a8ac6f34641a, queue=awx_private_queue) [2021-04-08 15:05:16,642: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(267ca291-c49e-4555-a03b-a8ac6f34641a, queue=awx_private_queue) 2021-04-08 15:05:16,969 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:04:08.817516+00:00 2021-04-08 15:05:16,982 DEBUG awx.main.dispatch delivered 267ca291-c49e-4555-a03b-a8ac6f34641a to worker[177] qsize 0 2021-04-08 15:05:16,985 DEBUG awx.main.dispatch task 267ca291-c49e-4555-a03b-a8ac6f34641a starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:05:16,986 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:05:16,657 DEBUG awx.main.dispatch task 584706e3-76dc-41f3-a55e-a0b03ca55811 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 15:05:17,001 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 15:05:17,053 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:05:17,115 DEBUG awx.main.dispatch task 267ca291-c49e-4555-a03b-a8ac6f34641a is finished 2021-04-08 15:05:17,419 DEBUG awx.main.dispatch task 584706e3-76dc-41f3-a55e-a0b03ca55811 is finished 2021-04-08 15:05:18,126 DEBUG awx.main.dispatch task 5d511f85-a93b-410f-8358-1f47adbb61ee is finished 2021-04-08 15:05:36,055 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(20eb7e56-eb72-4ed2-a3cb-ad5955094e83, queue=dev-awx-0) [2021-04-08 15:05:36,055: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(20eb7e56-eb72-4ed2-a3cb-ad5955094e83, queue=dev-awx-0) 2021-04-08 15:05:36,243 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f2ec634d-f310-44e2-9f40-4e5dc1a8e0b1, queue=dev-awx-0) [2021-04-08 15:05:36,243: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f2ec634d-f310-44e2-9f40-4e5dc1a8e0b1, queue=dev-awx-0) 2021-04-08 15:05:36,641 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3e1b5153-eb25-4184-8d3b-b6bf205e7bb1, queue=awx_private_queue) [2021-04-08 15:05:36,641: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3e1b5153-eb25-4184-8d3b-b6bf205e7bb1, queue=awx_private_queue) 2021-04-08 15:05:36,661 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dee6f4d3-e250-43b8-afe6-b6e903dab17e, queue=awx_private_queue) [2021-04-08 15:05:36,661: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dee6f4d3-e250-43b8-afe6-b6e903dab17e, queue=awx_private_queue) 2021-04-08 15:05:37,007 DEBUG awx.main.dispatch delivered 20eb7e56-eb72-4ed2-a3cb-ad5955094e83 to worker[176] qsize 0 2021-04-08 15:05:37,010 DEBUG awx.main.dispatch task 20eb7e56-eb72-4ed2-a3cb-ad5955094e83 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 15:05:37,011 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 15:05:37,010 DEBUG awx.main.dispatch delivered f2ec634d-f310-44e2-9f40-4e5dc1a8e0b1 to worker[177] qsize 0 2021-04-08 15:05:37,013 DEBUG awx.main.dispatch task f2ec634d-f310-44e2-9f40-4e5dc1a8e0b1 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:05:37,013 DEBUG awx.main.dispatch delivered 3e1b5153-eb25-4184-8d3b-b6bf205e7bb1 to worker[178] qsize 0 2021-04-08 15:05:37,016 DEBUG awx.main.dispatch delivered dee6f4d3-e250-43b8-afe6-b6e903dab17e to worker[180] qsize 0 2021-04-08 15:05:37,016 DEBUG awx.main.dispatch task 3e1b5153-eb25-4184-8d3b-b6bf205e7bb1 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:05:37,018 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:05:37,018 DEBUG awx.main.dispatch task dee6f4d3-e250-43b8-afe6-b6e903dab17e starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:05:37,355 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:05:37,383 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:05:16.963967+00:00 2021-04-08 15:05:38,185 DEBUG awx.main.dispatch task f2ec634d-f310-44e2-9f40-4e5dc1a8e0b1 is finished 2021-04-08 15:05:38,289 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:05:38,326 DEBUG awx.main.dispatch task 3e1b5153-eb25-4184-8d3b-b6bf205e7bb1 is finished 2021-04-08 15:05:38,327 DEBUG awx.main.dispatch task dee6f4d3-e250-43b8-afe6-b6e903dab17e is finished 2021-04-08 15:05:38,327 DEBUG awx.main.dispatch task 20eb7e56-eb72-4ed2-a3cb-ad5955094e83 is finished 2021-04-08 15:05:56,646 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(81f9bfb1-8d37-48ce-a034-a6c59bf55f8a, queue=awx_private_queue) [2021-04-08 15:05:56,646: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(81f9bfb1-8d37-48ce-a034-a6c59bf55f8a, queue=awx_private_queue) 2021-04-08 15:05:56,680 DEBUG awx.main.dispatch delivered 81f9bfb1-8d37-48ce-a034-a6c59bf55f8a to worker[177] qsize 0 2021-04-08 15:05:56,687 DEBUG awx.main.dispatch task 81f9bfb1-8d37-48ce-a034-a6c59bf55f8a starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:05:56,691 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:05:56,778 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:05:56,824 DEBUG awx.main.dispatch task 81f9bfb1-8d37-48ce-a034-a6c59bf55f8a is finished RESULT 2 OKREADY 2021-04-08 15:06:06,661 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(35e3b7db-3d84-45a0-9ad1-c399bb3744ca, queue=awx_private_queue) [2021-04-08 15:06:06,661: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(35e3b7db-3d84-45a0-9ad1-c399bb3744ca, queue=awx_private_queue) 2021-04-08 15:06:06,683 DEBUG awx.main.dispatch delivered 35e3b7db-3d84-45a0-9ad1-c399bb3744ca to worker[177] qsize 0 2021-04-08 15:06:06,686 DEBUG awx.main.dispatch task 35e3b7db-3d84-45a0-9ad1-c399bb3744ca starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:06:06,701 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:06:06,705 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:05:37.355988+00:00 2021-04-08 15:06:06,759 DEBUG awx.main.dispatch task 35e3b7db-3d84-45a0-9ad1-c399bb3744ca is finished 2021-04-08 15:06:16,642 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0c2949cc-a144-4cab-8f72-de5eada56640, queue=awx_private_queue) [2021-04-08 15:06:16,642: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0c2949cc-a144-4cab-8f72-de5eada56640, queue=awx_private_queue) 2021-04-08 15:06:16,885 DEBUG awx.main.dispatch delivered 0c2949cc-a144-4cab-8f72-de5eada56640 to worker[177] qsize 0 2021-04-08 15:06:16,887 DEBUG awx.main.dispatch task 0c2949cc-a144-4cab-8f72-de5eada56640 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:06:16,889 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:06:16,911 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:06:16,945 DEBUG awx.main.dispatch task 0c2949cc-a144-4cab-8f72-de5eada56640 is finished 2021-04-08 15:06:36,056 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8bc629f3-ca6a-4972-8ff9-90a7d5a4cd6c, queue=dev-awx-0) [2021-04-08 15:06:36,056: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8bc629f3-ca6a-4972-8ff9-90a7d5a4cd6c, queue=dev-awx-0) 2021-04-08 15:06:36,144 DEBUG awx.main.dispatch delivered 8bc629f3-ca6a-4972-8ff9-90a7d5a4cd6c to worker[177] qsize 0 2021-04-08 15:06:36,149 DEBUG awx.main.dispatch task 8bc629f3-ca6a-4972-8ff9-90a7d5a4cd6c starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 15:06:36,159 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 15:06:36,243 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(deb2e960-6694-40ef-b2e8-355a16bed7e8, queue=dev-awx-0) [2021-04-08 15:06:36,243: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(deb2e960-6694-40ef-b2e8-355a16bed7e8, queue=dev-awx-0) 2021-04-08 15:06:36,245 DEBUG awx.main.dispatch task 8bc629f3-ca6a-4972-8ff9-90a7d5a4cd6c is finished 2021-04-08 15:06:36,260 DEBUG awx.main.dispatch delivered deb2e960-6694-40ef-b2e8-355a16bed7e8 to worker[177] qsize 0 2021-04-08 15:06:36,262 DEBUG awx.main.dispatch task deb2e960-6694-40ef-b2e8-355a16bed7e8 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:06:36,276 DEBUG awx.main.dispatch task deb2e960-6694-40ef-b2e8-355a16bed7e8 is finished 2021-04-08 15:06:36,661 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e484ce37-4119-4ee7-892f-65035a36b713, queue=awx_private_queue) [2021-04-08 15:06:36,661: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e484ce37-4119-4ee7-892f-65035a36b713, queue=awx_private_queue) 2021-04-08 15:06:36,675 DEBUG awx.main.dispatch delivered e484ce37-4119-4ee7-892f-65035a36b713 to worker[176] qsize 0 2021-04-08 15:06:36,677 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(015555b4-d2d7-4992-a3c8-772d32f2a605, queue=awx_private_queue) [2021-04-08 15:06:36,677: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(015555b4-d2d7-4992-a3c8-772d32f2a605, queue=awx_private_queue) 2021-04-08 15:06:36,680 DEBUG awx.main.dispatch task e484ce37-4119-4ee7-892f-65035a36b713 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:06:36,690 DEBUG awx.main.dispatch delivered 015555b4-d2d7-4992-a3c8-772d32f2a605 to worker[177] qsize 0 2021-04-08 15:06:36,692 DEBUG awx.main.dispatch task 015555b4-d2d7-4992-a3c8-772d32f2a605 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:06:36,693 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:06:36,698 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:06:36,706 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:06:06.701722+00:00 2021-04-08 15:06:36,707 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:06:36,724 DEBUG awx.main.dispatch task e484ce37-4119-4ee7-892f-65035a36b713 is finished 2021-04-08 15:06:36,736 DEBUG awx.main.dispatch task 015555b4-d2d7-4992-a3c8-772d32f2a605 is finished 2021-04-08 15:06:56,677 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0c0c3b37-d60a-4635-9d91-30ed6740c149, queue=awx_private_queue) [2021-04-08 15:06:56,677: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0c0c3b37-d60a-4635-9d91-30ed6740c149, queue=awx_private_queue) 2021-04-08 15:06:56,698 DEBUG awx.main.dispatch delivered 0c0c3b37-d60a-4635-9d91-30ed6740c149 to worker[177] qsize 0 2021-04-08 15:06:56,700 DEBUG awx.main.dispatch task 0c0c3b37-d60a-4635-9d91-30ed6740c149 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:06:56,701 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:06:56,719 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:06:56,756 DEBUG awx.main.dispatch task 0c0c3b37-d60a-4635-9d91-30ed6740c149 is finished RESULT 2 OKREADY 2021-04-08 15:07:06,662 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2f490aea-68e0-49ba-81d6-6fc30e3b35d8, queue=awx_private_queue) [2021-04-08 15:07:06,662: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2f490aea-68e0-49ba-81d6-6fc30e3b35d8, queue=awx_private_queue) 2021-04-08 15:07:06,688 DEBUG awx.main.dispatch delivered 2f490aea-68e0-49ba-81d6-6fc30e3b35d8 to worker[178] qsize 0 2021-04-08 15:07:06,692 DEBUG awx.main.dispatch task 2f490aea-68e0-49ba-81d6-6fc30e3b35d8 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:07:07,045 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:07:07,064 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:06:36.698599+00:00 2021-04-08 15:07:07,177 DEBUG awx.main.dispatch task 2f490aea-68e0-49ba-81d6-6fc30e3b35d8 is finished 2021-04-08 15:07:16,677 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7cc9c140-e336-4d21-9f66-776f7db91795, queue=awx_private_queue) [2021-04-08 15:07:16,677: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7cc9c140-e336-4d21-9f66-776f7db91795, queue=awx_private_queue) 2021-04-08 15:07:17,287 DEBUG awx.main.dispatch delivered 7cc9c140-e336-4d21-9f66-776f7db91795 to worker[177] qsize 0 2021-04-08 15:07:17,289 DEBUG awx.main.dispatch task 7cc9c140-e336-4d21-9f66-776f7db91795 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:07:17,291 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:07:17,580 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:07:17,615 DEBUG awx.main.dispatch task 7cc9c140-e336-4d21-9f66-776f7db91795 is finished 2021-04-08 15:07:35,881 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(bf84532e-639f-49bb-9cae-334a3c168b7b, queue=dev-awx-0) [2021-04-08 15:07:35,881: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(bf84532e-639f-49bb-9cae-334a3c168b7b, queue=dev-awx-0) 2021-04-08 15:07:35,895 DEBUG awx.main.dispatch delivered bf84532e-639f-49bb-9cae-334a3c168b7b to worker[180] qsize 0 2021-04-08 15:07:35,899 DEBUG awx.main.dispatch task bf84532e-639f-49bb-9cae-334a3c168b7b starting awx.main.tasks.awx_isolated_heartbeat(*[]) 2021-04-08 15:07:35,904 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks. 2021-04-08 15:07:36,015 DEBUG awx.main.dispatch task bf84532e-639f-49bb-9cae-334a3c168b7b is finished 2021-04-08 15:07:36,055 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(51cd38d7-b92b-4ee5-86aa-74d5f068825e, queue=dev-awx-0) [2021-04-08 15:07:36,055: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(51cd38d7-b92b-4ee5-86aa-74d5f068825e, queue=dev-awx-0) 2021-04-08 15:07:36,082 DEBUG awx.main.dispatch delivered 51cd38d7-b92b-4ee5-86aa-74d5f068825e to worker[176] qsize 0 2021-04-08 15:07:36,084 DEBUG awx.main.dispatch task 51cd38d7-b92b-4ee5-86aa-74d5f068825e starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 15:07:36,086 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 15:07:36,220 DEBUG awx.main.dispatch task 51cd38d7-b92b-4ee5-86aa-74d5f068825e is finished 2021-04-08 15:07:36,243 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d43340e3-c743-423f-bc0e-48fc380a6ffa, queue=dev-awx-0) [2021-04-08 15:07:36,243: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d43340e3-c743-423f-bc0e-48fc380a6ffa, queue=dev-awx-0) 2021-04-08 15:07:36,259 DEBUG awx.main.dispatch delivered d43340e3-c743-423f-bc0e-48fc380a6ffa to worker[178] qsize 0 2021-04-08 15:07:36,263 DEBUG awx.main.dispatch task d43340e3-c743-423f-bc0e-48fc380a6ffa starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:07:36,276 DEBUG awx.main.dispatch task d43340e3-c743-423f-bc0e-48fc380a6ffa is finished 2021-04-08 15:07:36,662 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9f48f68b-1954-471c-9e66-a5eab6a91d08, queue=awx_private_queue) [2021-04-08 15:07:36,662: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9f48f68b-1954-471c-9e66-a5eab6a91d08, queue=awx_private_queue) 2021-04-08 15:07:36,685 DEBUG awx.main.dispatch delivered 9f48f68b-1954-471c-9e66-a5eab6a91d08 to worker[177] qsize 0 2021-04-08 15:07:36,688 DEBUG awx.main.dispatch task 9f48f68b-1954-471c-9e66-a5eab6a91d08 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:07:36,688 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(122d4b7d-b597-483d-8f20-e5266f55b9e3, queue=awx_private_queue) [2021-04-08 15:07:36,688: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(122d4b7d-b597-483d-8f20-e5266f55b9e3, queue=awx_private_queue) 2021-04-08 15:07:36,701 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:07:36,703 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:07:07.045830+00:00 2021-04-08 15:07:36,709 DEBUG awx.main.dispatch delivered 122d4b7d-b597-483d-8f20-e5266f55b9e3 to worker[176] qsize 0 2021-04-08 15:07:36,710 DEBUG awx.main.dispatch task 122d4b7d-b597-483d-8f20-e5266f55b9e3 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:07:36,713 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:07:36,716 DEBUG awx.main.dispatch task 9f48f68b-1954-471c-9e66-a5eab6a91d08 is finished 2021-04-08 15:07:36,729 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:07:36,761 DEBUG awx.main.dispatch task 122d4b7d-b597-483d-8f20-e5266f55b9e3 is finished 2021-04-08 15:07:56,705 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(747fc1ed-928f-48cd-ad83-4ab25f65ae0c, queue=awx_private_queue) [2021-04-08 15:07:56,705: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(747fc1ed-928f-48cd-ad83-4ab25f65ae0c, queue=awx_private_queue) 2021-04-08 15:07:56,734 DEBUG awx.main.dispatch delivered 747fc1ed-928f-48cd-ad83-4ab25f65ae0c to worker[176] qsize 0 2021-04-08 15:07:56,740 DEBUG awx.main.dispatch task 747fc1ed-928f-48cd-ad83-4ab25f65ae0c starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:07:56,744 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:08:01,170 DEBUG awx.main.scheduler Starting Scheduler RESULT 2 OKREADY 2021-04-08 15:08:01,294 DEBUG awx.main.dispatch task 747fc1ed-928f-48cd-ad83-4ab25f65ae0c is finished 2021-04-08 15:08:06,663 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(88061464-cb6f-4f0d-b447-32815fffdea8, queue=awx_private_queue) [2021-04-08 15:08:06,663: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(88061464-cb6f-4f0d-b447-32815fffdea8, queue=awx_private_queue) 2021-04-08 15:08:06,685 DEBUG awx.main.dispatch delivered 88061464-cb6f-4f0d-b447-32815fffdea8 to worker[176] qsize 0 2021-04-08 15:08:06,688 DEBUG awx.main.dispatch task 88061464-cb6f-4f0d-b447-32815fffdea8 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:08:06,707 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:08:06,711 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:07:36.701320+00:00 2021-04-08 15:08:06,762 DEBUG awx.main.dispatch task 88061464-cb6f-4f0d-b447-32815fffdea8 is finished 2021-04-08 15:08:16,700 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d6fa23a5-f612-4cc9-b968-a6796c5ff372, queue=awx_private_queue) [2021-04-08 15:08:16,700: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d6fa23a5-f612-4cc9-b968-a6796c5ff372, queue=awx_private_queue) 2021-04-08 15:08:16,948 DEBUG awx.main.dispatch delivered d6fa23a5-f612-4cc9-b968-a6796c5ff372 to worker[180] qsize 0 2021-04-08 15:08:16,952 DEBUG awx.main.dispatch task d6fa23a5-f612-4cc9-b968-a6796c5ff372 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:08:16,955 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:08:16,988 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:08:17,036 DEBUG awx.main.dispatch task d6fa23a5-f612-4cc9-b968-a6796c5ff372 is finished 2021-04-08 15:08:36,056 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e72ae61c-42c4-489b-90a9-8a7a19ae05f6, queue=dev-awx-0) [2021-04-08 15:08:36,056: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e72ae61c-42c4-489b-90a9-8a7a19ae05f6, queue=dev-awx-0) 2021-04-08 15:08:36,140 DEBUG awx.main.dispatch delivered e72ae61c-42c4-489b-90a9-8a7a19ae05f6 to worker[177] qsize 0 2021-04-08 15:08:36,144 DEBUG awx.main.dispatch task e72ae61c-42c4-489b-90a9-8a7a19ae05f6 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 15:08:36,148 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 15:08:36,243 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(07002dae-c60c-4572-a26c-9265eedfd9d5, queue=dev-awx-0) [2021-04-08 15:08:36,243: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(07002dae-c60c-4572-a26c-9265eedfd9d5, queue=dev-awx-0) 2021-04-08 15:08:36,259 DEBUG awx.main.dispatch delivered 07002dae-c60c-4572-a26c-9265eedfd9d5 to worker[178] qsize 0 2021-04-08 15:08:36,263 DEBUG awx.main.dispatch task e72ae61c-42c4-489b-90a9-8a7a19ae05f6 is finished 2021-04-08 15:08:36,265 DEBUG awx.main.dispatch task 07002dae-c60c-4572-a26c-9265eedfd9d5 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:08:36,285 DEBUG awx.main.dispatch task 07002dae-c60c-4572-a26c-9265eedfd9d5 is finished 2021-04-08 15:08:36,662 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bf958f53-3638-4e99-88fd-d3bdb86d46bc, queue=awx_private_queue) [2021-04-08 15:08:36,662: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bf958f53-3638-4e99-88fd-d3bdb86d46bc, queue=awx_private_queue) 2021-04-08 15:08:36,678 DEBUG awx.main.dispatch delivered bf958f53-3638-4e99-88fd-d3bdb86d46bc to worker[177] qsize 0 2021-04-08 15:08:36,680 DEBUG awx.main.dispatch task bf958f53-3638-4e99-88fd-d3bdb86d46bc starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:08:36,688 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:08:36,691 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:08:06.707978+00:00 2021-04-08 15:08:36,700 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a748bca7-8652-42a7-b8c2-edf53c163515, queue=awx_private_queue) [2021-04-08 15:08:36,700: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a748bca7-8652-42a7-b8c2-edf53c163515, queue=awx_private_queue) 2021-04-08 15:08:36,706 DEBUG awx.main.dispatch task bf958f53-3638-4e99-88fd-d3bdb86d46bc is finished 2021-04-08 15:08:36,720 DEBUG awx.main.dispatch delivered a748bca7-8652-42a7-b8c2-edf53c163515 to worker[180] qsize 0 2021-04-08 15:08:36,722 DEBUG awx.main.dispatch task a748bca7-8652-42a7-b8c2-edf53c163515 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:08:36,725 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:08:36,742 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:08:36,776 DEBUG awx.main.dispatch task a748bca7-8652-42a7-b8c2-edf53c163515 is finished 2021-04-08 15:08:56,700 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(374add3b-ef24-4073-a350-614ed5b2e72a, queue=awx_private_queue) [2021-04-08 15:08:56,700: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(374add3b-ef24-4073-a350-614ed5b2e72a, queue=awx_private_queue) 2021-04-08 15:08:56,717 DEBUG awx.main.dispatch delivered 374add3b-ef24-4073-a350-614ed5b2e72a to worker[177] qsize 0 2021-04-08 15:08:56,720 DEBUG awx.main.dispatch task 374add3b-ef24-4073-a350-614ed5b2e72a starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:08:56,721 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:08:56,998 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:08:57,049 DEBUG awx.main.dispatch task 374add3b-ef24-4073-a350-614ed5b2e72a is finished RESULT 2 OKREADY 2021-04-08 15:09:06,662 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b201d65e-4518-4b4d-bb20-d70b959cc412, queue=awx_private_queue) [2021-04-08 15:09:06,662: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b201d65e-4518-4b4d-bb20-d70b959cc412, queue=awx_private_queue) 2021-04-08 15:09:06,676 DEBUG awx.main.dispatch delivered b201d65e-4518-4b4d-bb20-d70b959cc412 to worker[176] qsize 0 2021-04-08 15:09:06,679 DEBUG awx.main.dispatch task b201d65e-4518-4b4d-bb20-d70b959cc412 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:09:07,549 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:09:07,597 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:08:36.688991+00:00 2021-04-08 15:09:07,662 DEBUG awx.main.dispatch task b201d65e-4518-4b4d-bb20-d70b959cc412 is finished 2021-04-08 15:09:16,700 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fd197553-8006-476b-9b82-de591c6eaaab, queue=awx_private_queue) [2021-04-08 15:09:16,700: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fd197553-8006-476b-9b82-de591c6eaaab, queue=awx_private_queue) 2021-04-08 15:09:16,718 DEBUG awx.main.dispatch delivered fd197553-8006-476b-9b82-de591c6eaaab to worker[178] qsize 0 2021-04-08 15:09:16,720 DEBUG awx.main.dispatch task fd197553-8006-476b-9b82-de591c6eaaab starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:09:16,721 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:09:16,742 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:09:16,823 DEBUG awx.main.dispatch task fd197553-8006-476b-9b82-de591c6eaaab is finished 2021-04-08 15:09:36,060 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6770fba0-0e1d-4f77-90a7-3f6e7156f188, queue=dev-awx-0) [2021-04-08 15:09:36,060: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6770fba0-0e1d-4f77-90a7-3f6e7156f188, queue=dev-awx-0) 2021-04-08 15:09:36,104 DEBUG awx.main.dispatch delivered 6770fba0-0e1d-4f77-90a7-3f6e7156f188 to worker[176] qsize 0 2021-04-08 15:09:36,108 DEBUG awx.main.dispatch task 6770fba0-0e1d-4f77-90a7-3f6e7156f188 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 15:09:36,111 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 15:09:36,175 DEBUG awx.main.dispatch task 6770fba0-0e1d-4f77-90a7-3f6e7156f188 is finished 2021-04-08 15:09:36,243 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5ea4a53a-fedb-4e41-94a6-dce609c2a6d2, queue=dev-awx-0) [2021-04-08 15:09:36,243: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5ea4a53a-fedb-4e41-94a6-dce609c2a6d2, queue=dev-awx-0) 2021-04-08 15:09:36,264 DEBUG awx.main.dispatch delivered 5ea4a53a-fedb-4e41-94a6-dce609c2a6d2 to worker[176] qsize 0 2021-04-08 15:09:36,456 DEBUG awx.main.dispatch task 5ea4a53a-fedb-4e41-94a6-dce609c2a6d2 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:09:36,469 DEBUG awx.main.dispatch task 5ea4a53a-fedb-4e41-94a6-dce609c2a6d2 is finished 2021-04-08 15:09:36,663 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(110add08-1926-4973-bd94-6e0c3c229178, queue=awx_private_queue) [2021-04-08 15:09:36,663: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(110add08-1926-4973-bd94-6e0c3c229178, queue=awx_private_queue) 2021-04-08 15:09:36,679 DEBUG awx.main.dispatch delivered 110add08-1926-4973-bd94-6e0c3c229178 to worker[178] qsize 0 2021-04-08 15:09:36,684 DEBUG awx.main.dispatch task 110add08-1926-4973-bd94-6e0c3c229178 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:09:36,700 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a25b02ee-7233-414e-b21a-b9ac95c93929, queue=awx_private_queue) [2021-04-08 15:09:36,700: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a25b02ee-7233-414e-b21a-b9ac95c93929, queue=awx_private_queue) 2021-04-08 15:09:36,702 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:09:36,708 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:09:07.549710+00:00 2021-04-08 15:09:36,715 DEBUG awx.main.dispatch delivered a25b02ee-7233-414e-b21a-b9ac95c93929 to worker[176] qsize 0 2021-04-08 15:09:36,717 DEBUG awx.main.dispatch task a25b02ee-7233-414e-b21a-b9ac95c93929 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:09:36,719 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:09:36,724 DEBUG awx.main.dispatch task 110add08-1926-4973-bd94-6e0c3c229178 is finished 2021-04-08 15:09:36,736 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:09:36,774 DEBUG awx.main.dispatch task a25b02ee-7233-414e-b21a-b9ac95c93929 is finished 2021-04-08 15:09:56,708 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f3a8fed7-95db-425d-bffa-22c54ae6a385, queue=awx_private_queue) [2021-04-08 15:09:56,708: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f3a8fed7-95db-425d-bffa-22c54ae6a385, queue=awx_private_queue) 2021-04-08 15:09:56,723 DEBUG awx.main.dispatch delivered f3a8fed7-95db-425d-bffa-22c54ae6a385 to worker[176] qsize 0 2021-04-08 15:09:56,725 DEBUG awx.main.dispatch task f3a8fed7-95db-425d-bffa-22c54ae6a385 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:09:56,727 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:09:56,745 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:09:56,783 DEBUG awx.main.dispatch task f3a8fed7-95db-425d-bffa-22c54ae6a385 is finished RESULT 2 OKREADY 2021-04-08 15:10:06,664 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5cd1c822-0184-47da-a596-4e08b05760e5, queue=awx_private_queue) [2021-04-08 15:10:06,664: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5cd1c822-0184-47da-a596-4e08b05760e5, queue=awx_private_queue) 2021-04-08 15:10:06,688 DEBUG awx.main.dispatch delivered 5cd1c822-0184-47da-a596-4e08b05760e5 to worker[176] qsize 0 2021-04-08 15:10:06,690 DEBUG awx.main.dispatch task 5cd1c822-0184-47da-a596-4e08b05760e5 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:10:06,702 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:10:06,705 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:09:36.702377+00:00 2021-04-08 15:10:06,756 DEBUG awx.main.dispatch task 5cd1c822-0184-47da-a596-4e08b05760e5 is finished 2021-04-08 15:10:16,708 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(899f8bfa-63b1-4512-95b2-f7a8b585af6b, queue=awx_private_queue) [2021-04-08 15:10:16,708: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(899f8bfa-63b1-4512-95b2-f7a8b585af6b, queue=awx_private_queue) 2021-04-08 15:10:16,733 DEBUG awx.main.dispatch delivered 899f8bfa-63b1-4512-95b2-f7a8b585af6b to worker[180] qsize 0 2021-04-08 15:10:16,737 DEBUG awx.main.dispatch task 899f8bfa-63b1-4512-95b2-f7a8b585af6b starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:10:16,742 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:10:16,766 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:10:16,794 DEBUG awx.main.dispatch task 899f8bfa-63b1-4512-95b2-f7a8b585af6b is finished 2021-04-08 15:10:36,062 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(3cd858b3-03d6-4ca8-92b6-99f433b85df1, queue=dev-awx-0) [2021-04-08 15:10:36,062: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(3cd858b3-03d6-4ca8-92b6-99f433b85df1, queue=dev-awx-0) 2021-04-08 15:10:36,111 DEBUG awx.main.dispatch delivered 3cd858b3-03d6-4ca8-92b6-99f433b85df1 to worker[176] qsize 0 2021-04-08 15:10:36,113 DEBUG awx.main.dispatch task 3cd858b3-03d6-4ca8-92b6-99f433b85df1 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 15:10:36,114 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 15:10:36,175 DEBUG awx.main.dispatch task 3cd858b3-03d6-4ca8-92b6-99f433b85df1 is finished 2021-04-08 15:10:36,243 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(86ff8f03-fbce-4778-8446-e2e9b0310942, queue=dev-awx-0) [2021-04-08 15:10:36,243: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(86ff8f03-fbce-4778-8446-e2e9b0310942, queue=dev-awx-0) 2021-04-08 15:10:36,268 DEBUG awx.main.dispatch delivered 86ff8f03-fbce-4778-8446-e2e9b0310942 to worker[176] qsize 0 2021-04-08 15:10:36,466 DEBUG awx.main.dispatch task 86ff8f03-fbce-4778-8446-e2e9b0310942 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:10:36,479 DEBUG awx.main.dispatch task 86ff8f03-fbce-4778-8446-e2e9b0310942 is finished 2021-04-08 15:10:36,662 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6f24f3cf-d1fa-45c0-8be5-d5881b013a5a, queue=awx_private_queue) [2021-04-08 15:10:36,662: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6f24f3cf-d1fa-45c0-8be5-d5881b013a5a, queue=awx_private_queue) 2021-04-08 15:10:36,680 DEBUG awx.main.dispatch delivered 6f24f3cf-d1fa-45c0-8be5-d5881b013a5a to worker[180] qsize 0 2021-04-08 15:10:36,683 DEBUG awx.main.dispatch task 6f24f3cf-d1fa-45c0-8be5-d5881b013a5a starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:10:36,694 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:10:36,697 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:10:06.702965+00:00 2021-04-08 15:10:36,708 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1f0f5fb2-d685-4fe2-9320-4c455d3eb1f1, queue=awx_private_queue) [2021-04-08 15:10:36,708: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1f0f5fb2-d685-4fe2-9320-4c455d3eb1f1, queue=awx_private_queue) 2021-04-08 15:10:36,710 DEBUG awx.main.dispatch task 6f24f3cf-d1fa-45c0-8be5-d5881b013a5a is finished 2021-04-08 15:10:36,732 DEBUG awx.main.dispatch delivered 1f0f5fb2-d685-4fe2-9320-4c455d3eb1f1 to worker[176] qsize 0 2021-04-08 15:10:36,741 DEBUG awx.main.dispatch task 1f0f5fb2-d685-4fe2-9320-4c455d3eb1f1 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:10:36,742 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:10:36,759 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:10:36,787 DEBUG awx.main.dispatch task 1f0f5fb2-d685-4fe2-9320-4c455d3eb1f1 is finished 2021-04-08 15:10:56,708 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a9a72fdf-a0db-41a5-bae3-e41ac448ee69, queue=awx_private_queue) [2021-04-08 15:10:56,708: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a9a72fdf-a0db-41a5-bae3-e41ac448ee69, queue=awx_private_queue) 2021-04-08 15:10:56,723 DEBUG awx.main.dispatch delivered a9a72fdf-a0db-41a5-bae3-e41ac448ee69 to worker[177] qsize 0 2021-04-08 15:10:56,729 DEBUG awx.main.dispatch task a9a72fdf-a0db-41a5-bae3-e41ac448ee69 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:10:56,737 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:10:56,773 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:10:56,820 DEBUG awx.main.dispatch task a9a72fdf-a0db-41a5-bae3-e41ac448ee69 is finished RESULT 2 OKREADY 2021-04-08 15:11:06,662 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6439f42b-b616-4455-b5ad-c34eb76536ca, queue=awx_private_queue) [2021-04-08 15:11:06,662: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6439f42b-b616-4455-b5ad-c34eb76536ca, queue=awx_private_queue) 2021-04-08 15:11:06,677 DEBUG awx.main.dispatch delivered 6439f42b-b616-4455-b5ad-c34eb76536ca to worker[176] qsize 0 2021-04-08 15:11:06,679 DEBUG awx.main.dispatch task 6439f42b-b616-4455-b5ad-c34eb76536ca starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:11:07,086 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:11:07,091 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:10:36.694291+00:00 2021-04-08 15:11:07,188 DEBUG awx.main.dispatch task 6439f42b-b616-4455-b5ad-c34eb76536ca is finished 2021-04-08 15:11:16,708 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c2279ec9-0fe6-4228-a638-6fa327d49b29, queue=awx_private_queue) [2021-04-08 15:11:16,708: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c2279ec9-0fe6-4228-a638-6fa327d49b29, queue=awx_private_queue) 2021-04-08 15:11:16,723 DEBUG awx.main.dispatch delivered c2279ec9-0fe6-4228-a638-6fa327d49b29 to worker[176] qsize 0 2021-04-08 15:11:16,725 DEBUG awx.main.dispatch task c2279ec9-0fe6-4228-a638-6fa327d49b29 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:11:16,726 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:11:16,933 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:11:16,970 DEBUG awx.main.dispatch task c2279ec9-0fe6-4228-a638-6fa327d49b29 is finished RESULT 2 OKREADY 2021-04-08 15:11:36,076 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(77c3c506-aece-4d4b-b78c-71ca0bc87b5d, queue=dev-awx-0) [2021-04-08 15:11:36,076: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(77c3c506-aece-4d4b-b78c-71ca0bc87b5d, queue=dev-awx-0) 2021-04-08 15:12:15,765 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8c1fae47-5f66-4f66-93bd-81ebf3cfefa8, queue=dev-awx-0) [2021-04-08 15:12:15,765: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8c1fae47-5f66-4f66-93bd-81ebf3cfefa8, queue=dev-awx-0) 2021-04-08 15:12:15,787 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f34796aa-3aaa-4fa3-bd6d-0d2b381e189b, queue=awx_private_queue) [2021-04-08 15:12:15,787: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f34796aa-3aaa-4fa3-bd6d-0d2b381e189b, queue=awx_private_queue) 2021-04-08 15:12:15,803 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c958cde5-54cc-410e-a025-908b7a59178b, queue=awx_private_queue) [2021-04-08 15:12:15,803: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c958cde5-54cc-410e-a025-908b7a59178b, queue=awx_private_queue) 2021-04-08 15:12:23,633 DEBUG awx.main.dispatch delivered 77c3c506-aece-4d4b-b78c-71ca0bc87b5d to worker[176] qsize 0 2021-04-08 15:12:23,641 DEBUG awx.main.dispatch delivered 8c1fae47-5f66-4f66-93bd-81ebf3cfefa8 to worker[178] qsize 0 2021-04-08 15:12:23,644 DEBUG awx.main.dispatch delivered f34796aa-3aaa-4fa3-bd6d-0d2b381e189b to worker[180] qsize 0 2021-04-08 15:12:23,646 DEBUG awx.main.dispatch delivered c958cde5-54cc-410e-a025-908b7a59178b to worker[177] qsize 0 2021-04-08 15:12:23,646 DEBUG awx.main.dispatch task 8c1fae47-5f66-4f66-93bd-81ebf3cfefa8 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:12:23,649 DEBUG awx.main.dispatch task f34796aa-3aaa-4fa3-bd6d-0d2b381e189b starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:12:23,650 DEBUG awx.main.dispatch task 77c3c506-aece-4d4b-b78c-71ca0bc87b5d starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 15:12:23,660 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 15:12:23,652 DEBUG awx.main.dispatch task c958cde5-54cc-410e-a025-908b7a59178b starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:12:23,660 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:12:25,199 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:12:25,327 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:12:25,361 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:11:07.087185+00:00 2021-04-08 15:12:25,378 DEBUG awx.main.dispatch task 77c3c506-aece-4d4b-b78c-71ca0bc87b5d is finished 2021-04-08 15:12:25,396 DEBUG awx.main.dispatch task 8c1fae47-5f66-4f66-93bd-81ebf3cfefa8 is finished 2021-04-08 15:12:25,498 DEBUG awx.main.dispatch task f34796aa-3aaa-4fa3-bd6d-0d2b381e189b is finished 2021-04-08 15:12:25,631 DEBUG awx.main.dispatch task c958cde5-54cc-410e-a025-908b7a59178b is finished 2021-04-08 15:12:35,807 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(74b6b9fd-b223-43ea-9f6e-4592795f1f66, queue=awx_private_queue) [2021-04-08 15:12:35,807: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(74b6b9fd-b223-43ea-9f6e-4592795f1f66, queue=awx_private_queue) 2021-04-08 15:12:35,823 DEBUG awx.main.dispatch delivered 74b6b9fd-b223-43ea-9f6e-4592795f1f66 to worker[180] qsize 0 2021-04-08 15:12:35,825 DEBUG awx.main.dispatch task 74b6b9fd-b223-43ea-9f6e-4592795f1f66 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:12:35,826 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:12:45,787 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(95874391-672f-4ffc-825a-dc15333e390e, queue=awx_private_queue) [2021-04-08 15:12:45,787: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(95874391-672f-4ffc-825a-dc15333e390e, queue=awx_private_queue) 2021-04-08 15:12:45,810 DEBUG awx.main.dispatch delivered 95874391-672f-4ffc-825a-dc15333e390e to worker[176] qsize 0 2021-04-08 15:12:45,812 DEBUG awx.main.dispatch task 95874391-672f-4ffc-825a-dc15333e390e starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:12:55,807 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(34a26736-e9f7-4bd2-ba43-c7092b2082b5, queue=awx_private_queue) [2021-04-08 15:12:55,807: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(34a26736-e9f7-4bd2-ba43-c7092b2082b5, queue=awx_private_queue) 2021-04-08 15:12:55,833 DEBUG awx.main.dispatch delivered 34a26736-e9f7-4bd2-ba43-c7092b2082b5 to worker[178] qsize 0 2021-04-08 15:12:55,834 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(586cf15f-dbf2-4830-af97-e1fee38ddcf2, queue=dev-awx-0) [2021-04-08 15:12:55,834: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(586cf15f-dbf2-4830-af97-e1fee38ddcf2, queue=dev-awx-0) 2021-04-08 15:12:55,835 DEBUG awx.main.dispatch task 34a26736-e9f7-4bd2-ba43-c7092b2082b5 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:12:55,837 DEBUG awx.main.scheduler Running Tower task manager. RESULT 2 OKREADY 2021-04-08 15:13:42,515 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:13:42,855 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:13:42,989 DEBUG awx.main.scheduler Not running scheduler, another task holds lock 2021-04-08 15:13:42,990 DEBUG awx.main.dispatch task 34a26736-e9f7-4bd2-ba43-c7092b2082b5 is finished 2021-04-08 15:13:43,015 DEBUG awx.main.dispatch task 74b6b9fd-b223-43ea-9f6e-4592795f1f66 is finished 2021-04-08 15:13:43,046 DEBUG awx.main.dispatch delivered 586cf15f-dbf2-4830-af97-e1fee38ddcf2 to worker[177] qsize 0 2021-04-08 15:13:43,224 DEBUG awx.main.dispatch task 586cf15f-dbf2-4830-af97-e1fee38ddcf2 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 15:13:43,228 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 15:13:50,938 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:12:25.327934+00:00 2021-04-08 15:13:15,770 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(6492cda6-1902-4d37-9325-8e50338abde1, queue=dev-awx-0) [2021-04-08 15:13:15,770: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(6492cda6-1902-4d37-9325-8e50338abde1, queue=dev-awx-0) 2021-04-08 15:13:52,594 DEBUG awx.main.dispatch delivered 6492cda6-1902-4d37-9325-8e50338abde1 to worker[178] qsize 0 2021-04-08 15:13:52,596 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b1cb368a-4ba8-418b-82d4-5d88149d3dab, queue=awx_private_queue) [2021-04-08 15:13:52,596: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b1cb368a-4ba8-418b-82d4-5d88149d3dab, queue=awx_private_queue) 2021-04-08 15:13:52,597 DEBUG awx.main.dispatch task 6492cda6-1902-4d37-9325-8e50338abde1 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:13:52,611 DEBUG awx.main.dispatch delivered b1cb368a-4ba8-418b-82d4-5d88149d3dab to worker[180] qsize 0 2021-04-08 15:13:52,612 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b418f659-5b7b-4601-8c01-581c1da08493, queue=awx_private_queue) [2021-04-08 15:13:52,612: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b418f659-5b7b-4601-8c01-581c1da08493, queue=awx_private_queue) 2021-04-08 15:13:52,612 DEBUG awx.main.dispatch task b1cb368a-4ba8-418b-82d4-5d88149d3dab starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:13:52,644 WARNING awx.main.dispatch scaling up worker pid:192 2021-04-08 15:13:52,650 DEBUG awx.main.dispatch delivered b418f659-5b7b-4601-8c01-581c1da08493 to worker[192] qsize 0 2021-04-08 15:13:52,658 DEBUG awx.main.dispatch task b418f659-5b7b-4601-8c01-581c1da08493 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:13:52,662 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:13:55,351 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:13:55,820 DEBUG awx.main.dispatch task 6492cda6-1902-4d37-9325-8e50338abde1 is finished 2021-04-08 15:13:55,834 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(21b70705-0ca7-4424-9611-39726454877f, queue=dev-awx-0) [2021-04-08 15:13:55,834: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(21b70705-0ca7-4424-9611-39726454877f, queue=dev-awx-0) 2021-04-08 15:13:55,851 WARNING awx.main.dispatch scaling down worker pid:178 2021-04-08 15:13:55,857 WARNING awx.main.dispatch worker exiting gracefully pid:178 2021-04-08 15:13:55,974 DEBUG awx.main.dispatch task 586cf15f-dbf2-4830-af97-e1fee38ddcf2 is finished RESULT 2 OKREADY 2021-04-08 15:14:06,243 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock 2021-04-08 15:14:06,243 DEBUG awx.main.dispatch task b1cb368a-4ba8-418b-82d4-5d88149d3dab is finished 2021-04-08 15:14:07,632 DEBUG awx.main.dispatch task 95874391-672f-4ffc-825a-dc15333e390e is finished 2021-04-08 15:14:12,617 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6c59385a-263e-4df3-a18b-50b8247a2847, queue=awx_private_queue) [2021-04-08 15:14:12,617: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6c59385a-263e-4df3-a18b-50b8247a2847, queue=awx_private_queue) 2021-04-08 15:14:16,910 DEBUG awx.main.dispatch delivered 21b70705-0ca7-4424-9611-39726454877f to worker[180] qsize 0 2021-04-08 15:14:16,912 DEBUG awx.main.dispatch delivered 6c59385a-263e-4df3-a18b-50b8247a2847 to worker[177] qsize 0 2021-04-08 15:14:16,912 DEBUG awx.main.dispatch task 21b70705-0ca7-4424-9611-39726454877f starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 15:14:16,914 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 15:14:16,914 DEBUG awx.main.dispatch task 6c59385a-263e-4df3-a18b-50b8247a2847 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:14:16,915 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:14:22,078 DEBUG awx.main.scheduler Not running scheduler, another task holds lock 2021-04-08 15:14:22,078 DEBUG awx.main.dispatch task 6c59385a-263e-4df3-a18b-50b8247a2847 is finished 2021-04-08 15:14:22,204 DEBUG awx.main.dispatch task b418f659-5b7b-4601-8c01-581c1da08493 is finished 2021-04-08 15:14:22,596 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(753f5036-d378-4fdf-9c4a-92ef730131e6, queue=awx_private_queue) [2021-04-08 15:14:22,596: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(753f5036-d378-4fdf-9c4a-92ef730131e6, queue=awx_private_queue) 2021-04-08 15:14:22,614 DEBUG awx.main.dispatch delivered 753f5036-d378-4fdf-9c4a-92ef730131e6 to worker[176] qsize 0 2021-04-08 15:14:22,616 DEBUG awx.main.dispatch task 753f5036-d378-4fdf-9c4a-92ef730131e6 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:14:22,654 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:14:22,660 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:13:42.516782+00:00 2021-04-08 15:14:23,165 DEBUG awx.main.dispatch task 21b70705-0ca7-4424-9611-39726454877f is finished 2021-04-08 15:14:23,372 DEBUG awx.main.dispatch task 753f5036-d378-4fdf-9c4a-92ef730131e6 is finished 2021-04-08 15:14:32,617 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9f3e676a-2e7c-4d88-9c17-e6a3270f9faa, queue=awx_private_queue) [2021-04-08 15:14:32,617: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9f3e676a-2e7c-4d88-9c17-e6a3270f9faa, queue=awx_private_queue) 2021-04-08 15:14:32,640 DEBUG awx.main.dispatch delivered 9f3e676a-2e7c-4d88-9c17-e6a3270f9faa to worker[192] qsize 0 2021-04-08 15:14:32,641 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d11f39e1-2cf6-4cc2-96ea-d92b5ee10e71, queue=dev-awx-0) [2021-04-08 15:14:32,641: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d11f39e1-2cf6-4cc2-96ea-d92b5ee10e71, queue=dev-awx-0) 2021-04-08 15:14:32,647 DEBUG awx.main.dispatch task 9f3e676a-2e7c-4d88-9c17-e6a3270f9faa starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:14:32,649 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:14:32,659 DEBUG awx.main.dispatch delivered d11f39e1-2cf6-4cc2-96ea-d92b5ee10e71 to worker[176] qsize 0 2021-04-08 15:14:32,662 DEBUG awx.main.dispatch task d11f39e1-2cf6-4cc2-96ea-d92b5ee10e71 starting awx.main.tasks.awx_k8s_reaper(*[]) RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY 2021-04-08 15:18:43,696 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:15:11,743 DEBUG awx.main.dispatch task d11f39e1-2cf6-4cc2-96ea-d92b5ee10e71 is finished 2021-04-08 15:18:44,773 DEBUG awx.main.dispatch task 9f3e676a-2e7c-4d88-9c17-e6a3270f9faa is finished 2021-04-08 15:14:52,611 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b5f3f865-0860-4521-966e-94de632b77fb, queue=awx_private_queue) [2021-04-08 15:14:52,611: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b5f3f865-0860-4521-966e-94de632b77fb, queue=awx_private_queue) 2021-04-08 15:18:45,078 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(74051b43-756f-4657-88db-64556fc9e8d8, queue=awx_private_queue) [2021-04-08 15:18:45,078: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(74051b43-756f-4657-88db-64556fc9e8d8, queue=awx_private_queue) 2021-04-08 15:18:45,083 DEBUG awx.main.dispatch delivered b5f3f865-0860-4521-966e-94de632b77fb to worker[177] qsize 0 2021-04-08 15:18:45,095 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(63e6e477-eaf9-40b1-8b58-b4e77495f38f, queue=dev-awx-0) [2021-04-08 15:18:45,095: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(63e6e477-eaf9-40b1-8b58-b4e77495f38f, queue=dev-awx-0) 2021-04-08 15:18:45,097 DEBUG awx.main.dispatch delivered 74051b43-756f-4657-88db-64556fc9e8d8 to worker[192] qsize 0 2021-04-08 15:18:45,101 DEBUG awx.main.dispatch task 74051b43-756f-4657-88db-64556fc9e8d8 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:18:45,103 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:18:45,099 DEBUG awx.main.dispatch task b5f3f865-0860-4521-966e-94de632b77fb starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:18:45,109 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(84df1e4e-dff8-439b-9677-570b797d35e1, queue=dev-awx-0) [2021-04-08 15:18:45,109: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(84df1e4e-dff8-439b-9677-570b797d35e1, queue=dev-awx-0) 2021-04-08 15:18:45,123 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(caa83ac7-60e4-4e22-a654-c80d006b870f, queue=dev-awx-0) [2021-04-08 15:18:45,123: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(caa83ac7-60e4-4e22-a654-c80d006b870f, queue=dev-awx-0) 2021-04-08 15:18:45,446 DEBUG awx.main.dispatch delivered 63e6e477-eaf9-40b1-8b58-b4e77495f38f to worker[180] qsize 0 2021-04-08 15:18:45,448 DEBUG awx.main.dispatch delivered 84df1e4e-dff8-439b-9677-570b797d35e1 to worker[176] qsize 0 2021-04-08 15:18:45,451 DEBUG awx.main.dispatch task 84df1e4e-dff8-439b-9677-570b797d35e1 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:18:45,451 DEBUG awx.main.dispatch task 63e6e477-eaf9-40b1-8b58-b4e77495f38f starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-04-08 15:18:45,457 DEBUG awx.main.tasks Cluster node heartbeat task. 2021-04-08 15:18:45,472 WARNING awx.main.dispatch scaling up worker pid:195 2021-04-08 15:18:45,478 DEBUG awx.main.dispatch delivered caa83ac7-60e4-4e22-a654-c80d006b870f to worker[195] qsize 0 2021-04-08 15:18:45,482 DEBUG awx.main.dispatch task caa83ac7-60e4-4e22-a654-c80d006b870f starting awx.main.tasks.awx_isolated_heartbeat(*[]) 2021-04-08 15:18:45,485 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks. 2021-04-08 15:18:47,661 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:18:47,692 DEBUG awx.main.tasks Last scheduler run was: 2021-04-08 15:14:22.654700+00:00 2021-04-08 15:18:48,150 DEBUG awx.main.dispatch task 84df1e4e-dff8-439b-9677-570b797d35e1 is finished 2021-04-08 15:18:48,174 DEBUG awx.main.scheduler Starting Scheduler 2021-04-08 15:18:48,190 DEBUG awx.main.dispatch task caa83ac7-60e4-4e22-a654-c80d006b870f is finished 2021-04-08 15:18:48,204 DEBUG awx.main.dispatch task b5f3f865-0860-4521-966e-94de632b77fb is finished 2021-04-08 15:18:48,273 DEBUG awx.main.dispatch task 74051b43-756f-4657-88db-64556fc9e8d8 is finished 2021-04-08 15:18:48,349 WARNING awx.main.tasks Rejoining the cluster as instance dev-awx-0. 2021-04-08 15:18:48,350 DEBUG awx.main.dispatch task 63e6e477-eaf9-40b1-8b58-b4e77495f38f is finished RESULT 2 OKREADY 2021-04-08 15:19:05,078 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(27b5d0f2-b1d2-4fbe-bfde-2146af245dd6, queue=awx_private_queue) [2021-04-08 15:19:05,078: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(27b5d0f2-b1d2-4fbe-bfde-2146af245dd6, queue=awx_private_queue) 2021-04-08 15:19:05,093 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0dd00cf6-b156-4b4a-8255-29006c42bfb7, queue=awx_private_queue) [2021-04-08 15:19:05,093: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0dd00cf6-b156-4b4a-8255-29006c42bfb7, queue=awx_private_queue) 2021-04-08 15:19:05,095 DEBUG awx.main.dispatch delivered 27b5d0f2-b1d2-4fbe-bfde-2146af245dd6 to worker[192] qsize 0 2021-04-08 15:19:05,099 DEBUG awx.main.dispatch task 27b5d0f2-b1d2-4fbe-bfde-2146af245dd6 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:19:05,101 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:19:05,108 DEBUG awx.main.dispatch delivered 0dd00cf6-b156-4b4a-8255-29006c42bfb7 to worker[180] qsize 0 2021-04-08 15:19:05,111 DEBUG awx.main.dispatch task 0dd00cf6-b156-4b4a-8255-29006c42bfb7 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:19:05,106 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection self.connect() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect self.connection = self.get_new_connection(conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection connection = Database.connect(**conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) psycopg2.OperationalError: FATAL: the database system is shutting down The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql cursor = self.connection.cursor() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor return self._cursor() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor self.ensure_connection() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection self.connect() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection self.connect() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect self.connection = self.get_new_connection(conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection connection = Database.connect(**conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) django.db.utils.OperationalError: FATAL: the database system is shutting down 2021-04-08 15:19:05,116 DEBUG awx.main.dispatch task 27b5d0f2-b1d2-4fbe-bfde-2146af245dd6 is finished 2021-04-08 15:19:05,117 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection self.connect() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect self.connection = self.get_new_connection(conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection connection = Database.connect(**conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) psycopg2.OperationalError: FATAL: the database system is shutting down The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired: File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock: File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock cursor = connections[using].cursor() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor return self._cursor() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor self.ensure_connection() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection self.connect() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection self.connect() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect self.connection = self.get_new_connection(conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection connection = Database.connect(**conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) django.db.utils.OperationalError: FATAL: the database system is shutting down 2021-04-08 15:19:05,123 DEBUG awx.main.dispatch task 0dd00cf6-b156-4b4a-8255-29006c42bfb7 is finished 2021-04-08 15:19:25,085 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(24b3a340-f1ba-49bd-99b0-a62c0fe56ee9, queue=awx_private_queue) [2021-04-08 15:19:25,085: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(24b3a340-f1ba-49bd-99b0-a62c0fe56ee9, queue=awx_private_queue) 2021-04-08 15:19:25,102 DEBUG awx.main.dispatch delivered 24b3a340-f1ba-49bd-99b0-a62c0fe56ee9 to worker[180] qsize 0 2021-04-08 15:19:25,107 DEBUG awx.main.dispatch task 24b3a340-f1ba-49bd-99b0-a62c0fe56ee9 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:19:25,109 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:19:35,095 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(317307e1-ac85-4d71-9368-d1040007f76f, queue=awx_private_queue) [2021-04-08 15:19:35,095: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(317307e1-ac85-4d71-9368-d1040007f76f, queue=awx_private_queue) 2021-04-08 15:19:35,115 DEBUG awx.main.dispatch delivered 317307e1-ac85-4d71-9368-d1040007f76f to worker[176] qsize 0 2021-04-08 15:19:35,118 DEBUG awx.main.dispatch task 317307e1-ac85-4d71-9368-d1040007f76f starting awx.main.tasks.awx_periodic_scheduler(*[]) RESULT 2 OKREADY RESULT 2 OKREADY 2021-04-08 15:21:35,794 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection self.connect() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect self.connection = self.get_new_connection(conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection connection = Database.connect(**conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) psycopg2.OperationalError: could not connect to server: Connection timed out Is the server running on host "awx-postgresql" (10.96.20.59) and accepting TCP/IP connections on port 5432? The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql cursor = self.connection.cursor() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor return self._cursor() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor self.ensure_connection() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection self.connect() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection self.connect() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect self.connection = self.get_new_connection(conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection connection = Database.connect(**conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) django.db.utils.OperationalError: could not connect to server: Connection timed out Is the server running on host "awx-postgresql" (10.96.20.59) and accepting TCP/IP connections on port 5432? 2021-04-08 15:21:35,916 DEBUG awx.main.dispatch task 24b3a340-f1ba-49bd-99b0-a62c0fe56ee9 is finished 2021-04-08 15:21:46,035 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection self.connect() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect self.connection = self.get_new_connection(conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection connection = Database.connect(**conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) psycopg2.OperationalError: could not connect to server: Connection timed out Is the server running on host "awx-postgresql" (10.96.20.59) and accepting TCP/IP connections on port 5432? The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired: File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock: File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock cursor = connections[using].cursor() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor return self._cursor() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor self.ensure_connection() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection self.connect() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection self.connect() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect self.connection = self.get_new_connection(conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection connection = Database.connect(**conn_params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) django.db.utils.OperationalError: could not connect to server: Connection timed out Is the server running on host "awx-postgresql" (10.96.20.59) and accepting TCP/IP connections on port 5432? 2021-04-08 15:21:46,741 DEBUG awx.main.dispatch task 317307e1-ac85-4d71-9368-d1040007f76f is finished 2021-04-08 15:19:45,085 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(eef84ae8-4d49-4839-86b4-4774f9090ff8, queue=awx_private_queue) [2021-04-08 15:19:45,085: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(eef84ae8-4d49-4839-86b4-4774f9090ff8, queue=awx_private_queue) 2021-04-08 15:21:56,319 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(a36c8786-c355-49bc-b06d-b5aa738e3d39, queue=dev-awx-0) [2021-04-08 15:21:56,319: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(a36c8786-c355-49bc-b06d-b5aa738e3d39, queue=dev-awx-0) 2021-04-08 15:21:56,319 DEBUG awx.main.dispatch delivered eef84ae8-4d49-4839-86b4-4774f9090ff8 to worker[177] qsize 0 2021-04-08 15:21:56,384 WARNING awx.main.dispatch scaling down worker pid:176 2021-04-08 15:21:56,395 WARNING awx.main.dispatch worker exiting gracefully pid:176 2021-04-08 15:21:56,387 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f40ac697-566b-475f-81a9-771227b42a96, queue=dev-awx-0) [2021-04-08 15:21:56,387: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f40ac697-566b-475f-81a9-771227b42a96, queue=dev-awx-0) 2021-04-08 15:21:56,382 DEBUG awx.main.dispatch task eef84ae8-4d49-4839-86b4-4774f9090ff8 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:21:56,419 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:21:56,422 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:21:56,427 DEBUG awx.main.dispatch task eef84ae8-4d49-4839-86b4-4774f9090ff8 is finished 2021-04-08 15:21:56,428 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(40b7262a-31c0-448c-929d-08188ddffcf8, queue=awx_private_queue) [2021-04-08 15:21:56,428: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(40b7262a-31c0-448c-929d-08188ddffcf8, queue=awx_private_queue) 2021-04-08 15:21:56,458 ERROR awx.main.dispatch failed to write inbound message Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write self.cleanup() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup reaper.reap(excluded_uuids=running_uuids) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap (changed, me) = Instance.objects.get_or_register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register return self.register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register if instance.exists(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists return self.query.has_results(using=self.db) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results return compiler.has_results() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results return bool(self.execute_sql(SINGLE)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ 2021-04-08 15:21:56,563 DEBUG awx.main.dispatch delivered f40ac697-566b-475f-81a9-771227b42a96 to worker[177] qsize 0 2021-04-08 15:21:56,575 DEBUG awx.main.dispatch delivered 40b7262a-31c0-448c-929d-08188ddffcf8 to worker[195] qsize 0 2021-04-08 15:21:56,572 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f590a87a-71b4-41df-95fe-0469299ed8bf, queue=awx_private_queue) [2021-04-08 15:21:56,572: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f590a87a-71b4-41df-95fe-0469299ed8bf, queue=awx_private_queue) 2021-04-08 15:21:56,576 DEBUG awx.main.dispatch task f40ac697-566b-475f-81a9-771227b42a96 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:21:56,597 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper for group in InstanceGroup.objects.filter(credential__isnull=False).iterator(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql cursor.close() psycopg2.errors.InvalidCursorName: cursor "_django_curs_140262630049344_12" does not exist 2021-04-08 15:21:56,601 DEBUG awx.main.dispatch delivered f590a87a-71b4-41df-95fe-0469299ed8bf to worker[192] qsize 0 2021-04-08 15:21:56,603 DEBUG awx.main.dispatch task f40ac697-566b-475f-81a9-771227b42a96 is finished 2021-04-08 15:21:56,581 DEBUG awx.main.dispatch task 40b7262a-31c0-448c-929d-08188ddffcf8 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:21:56,635 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:21:56,647 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:21:56,611 DEBUG awx.main.dispatch task f590a87a-71b4-41df-95fe-0469299ed8bf starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:21:56,660 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:21:56,657 DEBUG awx.main.dispatch task 40b7262a-31c0-448c-929d-08188ddffcf8 is finished 2021-04-08 15:21:56,662 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:21:56,679 DEBUG awx.main.dispatch task f590a87a-71b4-41df-95fe-0469299ed8bf is finished RESULT 2 OKREADY 2021-04-08 15:22:16,702 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3250c812-caf5-4350-879f-80f8e38fdd13, queue=awx_private_queue) [2021-04-08 15:22:16,702: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3250c812-caf5-4350-879f-80f8e38fdd13, queue=awx_private_queue) 2021-04-08 15:22:17,704 DEBUG awx.main.dispatch delivered 3250c812-caf5-4350-879f-80f8e38fdd13 to worker[195] qsize 0 2021-04-08 15:22:17,791 DEBUG awx.main.dispatch task 3250c812-caf5-4350-879f-80f8e38fdd13 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:22:17,882 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:22:17,886 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:22:17,892 DEBUG awx.main.dispatch task 3250c812-caf5-4350-879f-80f8e38fdd13 is finished 2021-04-08 15:22:28,512 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(902d8f68-77f3-4657-8b7c-4d40fe21adf1, queue=awx_private_queue) [2021-04-08 15:22:28,512: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(902d8f68-77f3-4657-8b7c-4d40fe21adf1, queue=awx_private_queue) 2021-04-08 15:22:29,759 DEBUG awx.main.dispatch delivered 902d8f68-77f3-4657-8b7c-4d40fe21adf1 to worker[195] qsize 0 2021-04-08 15:22:29,805 DEBUG awx.main.dispatch task 902d8f68-77f3-4657-8b7c-4d40fe21adf1 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:22:29,971 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:22:29,993 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:22:30,026 DEBUG awx.main.dispatch task 902d8f68-77f3-4657-8b7c-4d40fe21adf1 is finished 2021-04-08 15:22:36,626 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fc77cf2a-da55-4db3-b61f-01372eaab51a, queue=awx_private_queue) [2021-04-08 15:22:36,626: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fc77cf2a-da55-4db3-b61f-01372eaab51a, queue=awx_private_queue) 2021-04-08 15:22:37,286 DEBUG awx.main.dispatch delivered fc77cf2a-da55-4db3-b61f-01372eaab51a to worker[180] qsize 0 2021-04-08 15:22:37,329 DEBUG awx.main.dispatch task fc77cf2a-da55-4db3-b61f-01372eaab51a starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:22:37,481 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:22:37,486 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:22:37,508 DEBUG awx.main.dispatch task fc77cf2a-da55-4db3-b61f-01372eaab51a is finished 2021-04-08 15:22:56,329 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(179f779f-2e71-4906-8ea3-386a79963652, queue=dev-awx-0) [2021-04-08 15:22:56,329: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(179f779f-2e71-4906-8ea3-386a79963652, queue=dev-awx-0) 2021-04-08 15:22:59,364 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3b670930-1f32-49ee-bb14-b1947bb21057, queue=dev-awx-0) [2021-04-08 15:22:59,364: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3b670930-1f32-49ee-bb14-b1947bb21057, queue=dev-awx-0) 2021-04-08 15:22:59,413 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9eafcb15-290a-4e86-b1f4-50e0a5325383, queue=awx_private_queue) [2021-04-08 15:22:59,413: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9eafcb15-290a-4e86-b1f4-50e0a5325383, queue=awx_private_queue) 2021-04-08 15:22:59,501 ERROR awx.main.dispatch failed to write inbound message Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write self.cleanup() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup reaper.reap(excluded_uuids=running_uuids) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap (changed, me) = Instance.objects.get_or_register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register return self.register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register if instance.exists(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists return self.query.has_results(using=self.db) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results return compiler.has_results() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results return bool(self.execute_sql(SINGLE)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ 2021-04-08 15:22:59,775 DEBUG awx.main.dispatch delivered 3b670930-1f32-49ee-bb14-b1947bb21057 to worker[192] qsize 0 2021-04-08 15:22:59,688 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(21b583b2-8b61-46b2-b7ce-e1c694721f29, queue=awx_private_queue) [2021-04-08 15:22:59,688: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(21b583b2-8b61-46b2-b7ce-e1c694721f29, queue=awx_private_queue) 2021-04-08 15:22:59,780 DEBUG awx.main.dispatch delivered 9eafcb15-290a-4e86-b1f4-50e0a5325383 to worker[177] qsize 0 2021-04-08 15:22:59,799 DEBUG awx.main.dispatch delivered 21b583b2-8b61-46b2-b7ce-e1c694721f29 to worker[195] qsize 0 2021-04-08 15:22:59,797 DEBUG awx.main.dispatch task 9eafcb15-290a-4e86-b1f4-50e0a5325383 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:22:59,991 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:22:59,999 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:23:00,006 DEBUG awx.main.dispatch task 9eafcb15-290a-4e86-b1f4-50e0a5325383 is finished RESULT 2 OKREADY 2021-04-08 15:22:59,800 DEBUG awx.main.dispatch task 3b670930-1f32-49ee-bb14-b1947bb21057 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:23:00,094 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper for group in InstanceGroup.objects.filter(credential__isnull=False).iterator(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql cursor.close() psycopg2.errors.InvalidCursorName: cursor "_django_curs_140262630049344_1" does not exist 2021-04-08 15:23:00,105 DEBUG awx.main.dispatch task 3b670930-1f32-49ee-bb14-b1947bb21057 is finished 2021-04-08 15:23:00,132 DEBUG awx.main.dispatch task 21b583b2-8b61-46b2-b7ce-e1c694721f29 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:23:00,339 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:23:00,370 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:23:00,381 DEBUG awx.main.dispatch task 21b583b2-8b61-46b2-b7ce-e1c694721f29 is finished 2021-04-08 15:23:19,428 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(90a26188-2dbf-4a58-9841-25b5357e3b55, queue=awx_private_queue) [2021-04-08 15:23:19,428: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(90a26188-2dbf-4a58-9841-25b5357e3b55, queue=awx_private_queue) 2021-04-08 15:23:19,540 DEBUG awx.main.dispatch delivered 90a26188-2dbf-4a58-9841-25b5357e3b55 to worker[180] qsize 0 2021-04-08 15:23:19,552 DEBUG awx.main.dispatch task 90a26188-2dbf-4a58-9841-25b5357e3b55 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:23:19,692 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:23:19,699 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:23:19,721 DEBUG awx.main.dispatch task 90a26188-2dbf-4a58-9841-25b5357e3b55 is finished 2021-04-08 15:23:29,580 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c14c48c1-7eb5-413a-bb04-6370a05002b7, queue=awx_private_queue) [2021-04-08 15:23:29,580: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c14c48c1-7eb5-413a-bb04-6370a05002b7, queue=awx_private_queue) 2021-04-08 15:23:29,613 DEBUG awx.main.dispatch delivered c14c48c1-7eb5-413a-bb04-6370a05002b7 to worker[177] qsize 0 2021-04-08 15:23:29,634 DEBUG awx.main.dispatch task c14c48c1-7eb5-413a-bb04-6370a05002b7 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:23:29,768 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:23:29,777 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:23:29,801 DEBUG awx.main.dispatch task c14c48c1-7eb5-413a-bb04-6370a05002b7 is finished 2021-04-08 15:23:39,432 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(49f59d14-a591-456b-ba53-38c265829174, queue=awx_private_queue) [2021-04-08 15:23:39,432: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(49f59d14-a591-456b-ba53-38c265829174, queue=awx_private_queue) 2021-04-08 15:23:39,546 DEBUG awx.main.dispatch delivered 49f59d14-a591-456b-ba53-38c265829174 to worker[195] qsize 0 2021-04-08 15:23:39,575 DEBUG awx.main.dispatch task 49f59d14-a591-456b-ba53-38c265829174 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:23:39,613 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:23:39,619 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:23:39,625 DEBUG awx.main.dispatch task 49f59d14-a591-456b-ba53-38c265829174 is finished 2021-04-08 15:23:56,867 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(7860348f-42e8-4ba8-86c2-12f0f432e29b, queue=dev-awx-0) [2021-04-08 15:23:56,867: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(7860348f-42e8-4ba8-86c2-12f0f432e29b, queue=dev-awx-0) 2021-04-08 15:23:57,599 ERROR awx.main.dispatch failed to write inbound message Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write self.cleanup() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup reaper.reap(excluded_uuids=running_uuids) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap (changed, me) = Instance.objects.get_or_register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register return self.register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register if instance.exists(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists return self.query.has_results(using=self.db) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results return compiler.has_results() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results return bool(self.execute_sql(SINGLE)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ 2021-04-08 15:23:58,014 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(fa279f74-817c-4037-a9cd-520727aa8060, queue=dev-awx-0) [2021-04-08 15:23:58,014: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(fa279f74-817c-4037-a9cd-520727aa8060, queue=dev-awx-0) 2021-04-08 15:23:58,307 DEBUG awx.main.dispatch delivered fa279f74-817c-4037-a9cd-520727aa8060 to worker[192] qsize 0 2021-04-08 15:23:58,318 DEBUG awx.main.dispatch task fa279f74-817c-4037-a9cd-520727aa8060 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:23:58,371 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper for group in InstanceGroup.objects.filter(credential__isnull=False).iterator(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql cursor.close() psycopg2.errors.InvalidCursorName: cursor "_django_curs_140262630049344_2" does not exist 2021-04-08 15:23:58,379 DEBUG awx.main.dispatch task fa279f74-817c-4037-a9cd-520727aa8060 is finished 2021-04-08 15:23:59,418 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5e476461-0b9d-435f-8512-62f29a375707, queue=awx_private_queue) [2021-04-08 15:23:59,418: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5e476461-0b9d-435f-8512-62f29a375707, queue=awx_private_queue) 2021-04-08 15:23:59,476 DEBUG awx.main.dispatch delivered 5e476461-0b9d-435f-8512-62f29a375707 to worker[195] qsize 0 2021-04-08 15:23:59,488 DEBUG awx.main.dispatch task 5e476461-0b9d-435f-8512-62f29a375707 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:23:59,533 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:23:59,536 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:23:59,544 DEBUG awx.main.dispatch task 5e476461-0b9d-435f-8512-62f29a375707 is finished 2021-04-08 15:23:59,948 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(703ead6d-df4e-4155-90d3-2128929768ab, queue=awx_private_queue) [2021-04-08 15:23:59,948: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(703ead6d-df4e-4155-90d3-2128929768ab, queue=awx_private_queue) RESULT 2 OKREADY 2021-04-08 15:24:01,558 DEBUG awx.main.dispatch delivered 703ead6d-df4e-4155-90d3-2128929768ab to worker[192] qsize 0 2021-04-08 15:24:01,612 DEBUG awx.main.dispatch task 703ead6d-df4e-4155-90d3-2128929768ab starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:24:01,702 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:24:01,716 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:24:01,722 DEBUG awx.main.dispatch task 703ead6d-df4e-4155-90d3-2128929768ab is finished 2021-04-08 15:24:30,611 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0604327a-6524-4809-b27d-f7279f480ff4, queue=awx_private_queue) [2021-04-08 15:24:30,611: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0604327a-6524-4809-b27d-f7279f480ff4, queue=awx_private_queue) 2021-04-08 15:24:32,029 DEBUG awx.main.dispatch delivered 0604327a-6524-4809-b27d-f7279f480ff4 to worker[180] qsize 0 2021-04-08 15:24:32,032 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e97b8371-b24d-41bf-ad1e-7ad407b9eece, queue=awx_private_queue) [2021-04-08 15:24:32,032: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e97b8371-b24d-41bf-ad1e-7ad407b9eece, queue=awx_private_queue) 2021-04-08 15:24:32,475 DEBUG awx.main.dispatch delivered e97b8371-b24d-41bf-ad1e-7ad407b9eece to worker[177] qsize 0 2021-04-08 15:24:32,060 DEBUG awx.main.dispatch task 0604327a-6524-4809-b27d-f7279f480ff4 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:24:34,968 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:24:35,007 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:24:35,062 DEBUG awx.main.dispatch task 0604327a-6524-4809-b27d-f7279f480ff4 is finished 2021-04-08 15:24:32,643 DEBUG awx.main.dispatch task e97b8371-b24d-41bf-ad1e-7ad407b9eece starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:24:36,047 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:24:36,103 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:24:36,134 DEBUG awx.main.dispatch task e97b8371-b24d-41bf-ad1e-7ad407b9eece is finished 2021-04-08 15:24:39,421 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(34c249dd-bcf4-449a-9b47-abe1e3a0a0c3, queue=awx_private_queue) [2021-04-08 15:24:39,421: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(34c249dd-bcf4-449a-9b47-abe1e3a0a0c3, queue=awx_private_queue) 2021-04-08 15:24:44,545 DEBUG awx.main.dispatch delivered 34c249dd-bcf4-449a-9b47-abe1e3a0a0c3 to worker[192] qsize 0 2021-04-08 15:24:56,372 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(fab8bc42-0492-4cc1-ba33-27216d031062, queue=dev-awx-0) [2021-04-08 15:24:56,372: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(fab8bc42-0492-4cc1-ba33-27216d031062, queue=dev-awx-0) 2021-04-08 15:24:44,609 DEBUG awx.main.dispatch task 34c249dd-bcf4-449a-9b47-abe1e3a0a0c3 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:24:57,316 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:24:57,371 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:24:57,381 DEBUG awx.main.dispatch task 34c249dd-bcf4-449a-9b47-abe1e3a0a0c3 is finished RESULT 2 OKREADY 2021-04-08 15:24:58,517 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(1bf13fd2-a596-40a2-8650-4e65cbf06548, queue=dev-awx-0) [2021-04-08 15:24:58,517: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(1bf13fd2-a596-40a2-8650-4e65cbf06548, queue=dev-awx-0) 2021-04-08 15:24:56,528 ERROR awx.main.dispatch failed to write inbound message Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write self.cleanup() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup reaper.reap(excluded_uuids=running_uuids) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap (changed, me) = Instance.objects.get_or_register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register return self.register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register if instance.exists(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists return self.query.has_results(using=self.db) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results return compiler.has_results() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results return bool(self.execute_sql(SINGLE)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ 2021-04-08 15:25:06,825 DEBUG awx.main.dispatch delivered 1bf13fd2-a596-40a2-8650-4e65cbf06548 to worker[192] qsize 0 2021-04-08 15:25:06,778 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0c64e3c0-3d9b-43af-bd7f-2cfec31a0859, queue=awx_private_queue) [2021-04-08 15:25:06,778: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0c64e3c0-3d9b-43af-bd7f-2cfec31a0859, queue=awx_private_queue) 2021-04-08 15:25:06,854 DEBUG awx.main.dispatch delivered 0c64e3c0-3d9b-43af-bd7f-2cfec31a0859 to worker[177] qsize 0 2021-04-08 15:25:06,856 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6958fd6b-0218-4382-aaba-7ca2a87af7e5, queue=awx_private_queue) [2021-04-08 15:25:06,856: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6958fd6b-0218-4382-aaba-7ca2a87af7e5, queue=awx_private_queue) 2021-04-08 15:25:06,863 DEBUG awx.main.dispatch task 1bf13fd2-a596-40a2-8650-4e65cbf06548 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:25:06,863 DEBUG awx.main.dispatch task 0c64e3c0-3d9b-43af-bd7f-2cfec31a0859 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:25:18,713 DEBUG awx.main.dispatch delivered 6958fd6b-0218-4382-aaba-7ca2a87af7e5 to worker[180] qsize 0 2021-04-08 15:25:18,723 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:25:18,719 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper for group in InstanceGroup.objects.filter(credential__isnull=False).iterator(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql cursor.close() psycopg2.errors.InvalidCursorName: cursor "_django_curs_140262630049344_3" does not exist 2021-04-08 15:25:18,731 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:25:18,724 DEBUG awx.main.dispatch task 6958fd6b-0218-4382-aaba-7ca2a87af7e5 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:25:18,744 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:25:18,730 DEBUG awx.main.dispatch task 1bf13fd2-a596-40a2-8650-4e65cbf06548 is finished 2021-04-08 15:25:18,741 DEBUG awx.main.dispatch task 0c64e3c0-3d9b-43af-bd7f-2cfec31a0859 is finished 2021-04-08 15:25:18,747 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:25:18,753 DEBUG awx.main.dispatch task 6958fd6b-0218-4382-aaba-7ca2a87af7e5 is finished RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY 2021-04-08 15:26:12,798 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bf5fd6ab-596b-47fd-8264-a95fa7b56d6d, queue=awx_private_queue) [2021-04-08 15:26:12,798: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bf5fd6ab-596b-47fd-8264-a95fa7b56d6d, queue=awx_private_queue) 2021-04-08 15:29:46,337 DEBUG awx.main.dispatch delivered bf5fd6ab-596b-47fd-8264-a95fa7b56d6d to worker[195] qsize 0 RESULT 2 OKREADY 2021-04-08 15:29:50,437 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2f7da9b1-4c3f-42f4-a83c-19ac9a19a99e, queue=awx_private_queue) [2021-04-08 15:29:50,437: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2f7da9b1-4c3f-42f4-a83c-19ac9a19a99e, queue=awx_private_queue) 2021-04-08 15:30:20,203 DEBUG awx.main.dispatch delivered 2f7da9b1-4c3f-42f4-a83c-19ac9a19a99e to worker[180] qsize 0 2021-04-08 15:30:20,527 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(fceae225-412e-4287-86dd-3cedc4336357, queue=dev-awx-0) [2021-04-08 15:30:20,527: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(fceae225-412e-4287-86dd-3cedc4336357, queue=dev-awx-0) 2021-04-08 15:30:20,727 ERROR awx.main.dispatch failed to write inbound message Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write self.cleanup() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup reaper.reap(excluded_uuids=running_uuids) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap (changed, me) = Instance.objects.get_or_register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register return self.register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register if instance.exists(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists return self.query.has_results(using=self.db) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results return compiler.has_results() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results return bool(self.execute_sql(SINGLE)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ 2021-04-08 15:30:20,686 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2813ff7a-1ea9-4749-8f97-1577791e6d9e, queue=dev-awx-0) [2021-04-08 15:30:20,686: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2813ff7a-1ea9-4749-8f97-1577791e6d9e, queue=dev-awx-0) 2021-04-08 15:30:20,702 DEBUG awx.main.dispatch task 2f7da9b1-4c3f-42f4-a83c-19ac9a19a99e starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:30:20,894 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:30:20,896 DEBUG awx.main.dispatch delivered 2813ff7a-1ea9-4749-8f97-1577791e6d9e to worker[177] qsize 0 2021-04-08 15:30:20,911 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:30:20,957 DEBUG awx.main.dispatch task 2f7da9b1-4c3f-42f4-a83c-19ac9a19a99e is finished 2021-04-08 15:29:50,433 DEBUG awx.main.dispatch task bf5fd6ab-596b-47fd-8264-a95fa7b56d6d starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:30:22,053 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:30:20,912 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(cce9a9e9-41b7-46f6-a61f-009cb92c2ef6, queue=dev-awx-0) [2021-04-08 15:30:20,912: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(cce9a9e9-41b7-46f6-a61f-009cb92c2ef6, queue=dev-awx-0) 2021-04-08 15:30:23,108 DEBUG awx.main.dispatch delivered cce9a9e9-41b7-46f6-a61f-009cb92c2ef6 to worker[192] qsize 0 2021-04-08 15:30:23,828 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:30:24,149 DEBUG awx.main.dispatch task bf5fd6ab-596b-47fd-8264-a95fa7b56d6d is finished 2021-04-08 15:30:23,121 DEBUG awx.main.dispatch task cce9a9e9-41b7-46f6-a61f-009cb92c2ef6 starting awx.main.tasks.awx_isolated_heartbeat(*[]) 2021-04-08 15:30:28,865 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks. 2021-04-08 15:30:28,887 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instance" does not exist LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 470, in awx_isolated_heartbeat for isolated_instance in isolated_instance_qs: File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instance" does not exist LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst... ^ 2021-04-08 15:30:28,898 DEBUG awx.main.dispatch task cce9a9e9-41b7-46f6-a61f-009cb92c2ef6 is finished 2021-04-08 15:30:20,950 DEBUG awx.main.dispatch task 2813ff7a-1ea9-4749-8f97-1577791e6d9e starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:30:28,947 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper for group in InstanceGroup.objects.filter(credential__isnull=False).iterator(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql cursor.close() psycopg2.errors.InvalidCursorName: cursor "_django_curs_140262630049344_13" does not exist 2021-04-08 15:30:28,954 DEBUG awx.main.dispatch task 2813ff7a-1ea9-4749-8f97-1577791e6d9e is finished 2021-04-08 15:30:24,350 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(008531b8-869a-4e6f-ad8c-0fb467acdc77, queue=awx_private_queue) [2021-04-08 15:30:24,350: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(008531b8-869a-4e6f-ad8c-0fb467acdc77, queue=awx_private_queue) 2021-04-08 15:30:38,400 DEBUG awx.main.dispatch delivered 008531b8-869a-4e6f-ad8c-0fb467acdc77 to worker[192] qsize 0 2021-04-08 15:30:38,420 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b07d1021-2ae4-442d-adf4-e448e337467f, queue=awx_private_queue) [2021-04-08 15:30:38,420: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b07d1021-2ae4-442d-adf4-e448e337467f, queue=awx_private_queue) 2021-04-08 15:30:38,660 DEBUG awx.main.dispatch delivered b07d1021-2ae4-442d-adf4-e448e337467f to worker[180] qsize 0 2021-04-08 15:30:38,676 DEBUG awx.main.dispatch task 008531b8-869a-4e6f-ad8c-0fb467acdc77 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:30:38,676 DEBUG awx.main.dispatch task b07d1021-2ae4-442d-adf4-e448e337467f starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:30:38,806 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:30:38,811 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:30:38,808 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:30:38,814 DEBUG awx.main.dispatch task b07d1021-2ae4-442d-adf4-e448e337467f is finished 2021-04-08 15:30:38,815 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:30:38,823 DEBUG awx.main.dispatch task 008531b8-869a-4e6f-ad8c-0fb467acdc77 is finished RESULT 2 OKREADY 2021-04-08 15:31:11,790 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(07e416e3-2249-419c-a4be-e32d21a33eec, queue=awx_private_queue) [2021-04-08 15:31:11,790: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(07e416e3-2249-419c-a4be-e32d21a33eec, queue=awx_private_queue) 2021-04-08 15:31:55,384 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d2e25c5c-9afe-4aa5-b16a-11101ea4a6a0, queue=awx_private_queue) [2021-04-08 15:31:55,384: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d2e25c5c-9afe-4aa5-b16a-11101ea4a6a0, queue=awx_private_queue) 2021-04-08 15:31:55,375 DEBUG awx.main.dispatch delivered 07e416e3-2249-419c-a4be-e32d21a33eec to worker[192] qsize 0 2021-04-08 15:31:56,747 DEBUG awx.main.dispatch delivered d2e25c5c-9afe-4aa5-b16a-11101ea4a6a0 to worker[180] qsize 0 2021-04-08 15:31:56,756 DEBUG awx.main.dispatch task 07e416e3-2249-419c-a4be-e32d21a33eec starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:31:56,901 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:31:56,904 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:31:56,910 DEBUG awx.main.dispatch task 07e416e3-2249-419c-a4be-e32d21a33eec is finished 2021-04-08 15:31:56,887 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(aa3a884f-ab06-444e-ae0f-53b756c17436, queue=dev-awx-0) [2021-04-08 15:31:56,887: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(aa3a884f-ab06-444e-ae0f-53b756c17436, queue=dev-awx-0) 2021-04-08 15:31:57,337 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a3d0509c-cb47-4eb2-9e9e-82ce5406b5c6, queue=dev-awx-0) [2021-04-08 15:31:57,337: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a3d0509c-cb47-4eb2-9e9e-82ce5406b5c6, queue=dev-awx-0) 2021-04-08 15:31:56,829 DEBUG awx.main.dispatch task d2e25c5c-9afe-4aa5-b16a-11101ea4a6a0 starting awx.main.tasks.awx_periodic_scheduler(*[]) RESULT 2 OKREADY 2021-04-08 15:32:01,101 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:32:01,106 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:32:01,112 DEBUG awx.main.dispatch task d2e25c5c-9afe-4aa5-b16a-11101ea4a6a0 is finished 2021-04-08 15:31:57,691 ERROR awx.main.dispatch failed to write inbound message Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write self.cleanup() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup reaper.reap(excluded_uuids=running_uuids) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap (changed, me) = Instance.objects.get_or_register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register return self.register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register if instance.exists(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists return self.query.has_results(using=self.db) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results return compiler.has_results() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results return bool(self.execute_sql(SINGLE)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ 2021-04-08 15:31:57,792 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(665e3a09-e5db-4413-96e5-24ae3b9dbb88, queue=awx_private_queue) [2021-04-08 15:31:57,792: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(665e3a09-e5db-4413-96e5-24ae3b9dbb88, queue=awx_private_queue) 2021-04-08 15:32:01,123 DEBUG awx.main.dispatch delivered a3d0509c-cb47-4eb2-9e9e-82ce5406b5c6 to worker[180] qsize 0 2021-04-08 15:32:01,140 DEBUG awx.main.dispatch delivered 665e3a09-e5db-4413-96e5-24ae3b9dbb88 to worker[177] qsize 0 2021-04-08 15:32:01,158 DEBUG awx.main.dispatch task a3d0509c-cb47-4eb2-9e9e-82ce5406b5c6 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:32:01,399 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper for group in InstanceGroup.objects.filter(credential__isnull=False).iterator(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql cursor.close() psycopg2.errors.InvalidCursorName: cursor "_django_curs_140262630049344_7" does not exist 2021-04-08 15:32:01,408 DEBUG awx.main.dispatch task a3d0509c-cb47-4eb2-9e9e-82ce5406b5c6 is finished 2021-04-08 15:32:01,169 DEBUG awx.main.dispatch task 665e3a09-e5db-4413-96e5-24ae3b9dbb88 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:32:01,465 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:32:01,482 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:32:01,490 DEBUG awx.main.dispatch task 665e3a09-e5db-4413-96e5-24ae3b9dbb88 is finished RESULT 2 OKREADY 2021-04-08 15:32:46,320 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(391ea2fe-f6dd-483f-afdc-b84734f83eeb, queue=awx_private_queue) [2021-04-08 15:32:46,320: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(391ea2fe-f6dd-483f-afdc-b84734f83eeb, queue=awx_private_queue) 2021-04-08 15:33:43,576 DEBUG awx.main.dispatch delivered 391ea2fe-f6dd-483f-afdc-b84734f83eeb to worker[180] qsize 0 2021-04-08 15:33:49,084 DEBUG awx.main.dispatch task 391ea2fe-f6dd-483f-afdc-b84734f83eeb starting awx.main.scheduler.tasks.run_task_manager(*[]) RESULT 2 OKREADY 2021-04-08 15:34:01,222 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:34:01,229 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:34:01,235 DEBUG awx.main.dispatch task 391ea2fe-f6dd-483f-afdc-b84734f83eeb is finished 2021-04-08 15:33:45,612 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1d1bc366-6dc7-45eb-a4b5-d20ab11bbeeb, queue=awx_private_queue) [2021-04-08 15:33:45,612: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1d1bc366-6dc7-45eb-a4b5-d20ab11bbeeb, queue=awx_private_queue) 2021-04-08 15:34:01,493 DEBUG awx.main.dispatch delivered 1d1bc366-6dc7-45eb-a4b5-d20ab11bbeeb to worker[192] qsize 0 2021-04-08 15:34:01,548 DEBUG awx.main.dispatch task 1d1bc366-6dc7-45eb-a4b5-d20ab11bbeeb starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:34:01,743 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:34:01,755 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:34:01,761 DEBUG awx.main.dispatch task 1d1bc366-6dc7-45eb-a4b5-d20ab11bbeeb is finished 2021-04-08 15:34:01,501 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(da8a8778-3990-420d-aeaf-4fcbef2d778e, queue=dev-awx-0) [2021-04-08 15:34:01,501: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(da8a8778-3990-420d-aeaf-4fcbef2d778e, queue=dev-awx-0) 2021-04-08 15:34:01,814 ERROR awx.main.dispatch failed to write inbound message Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write self.cleanup() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup reaper.reap(excluded_uuids=running_uuids) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap (changed, me) = Instance.objects.get_or_register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register return self.register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register if instance.exists(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists return self.query.has_results(using=self.db) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results return compiler.has_results() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results return bool(self.execute_sql(SINGLE)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ 2021-04-08 15:34:01,793 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(e26e3a12-8e37-4b30-aa45-45b26888524c, queue=dev-awx-0) [2021-04-08 15:34:01,793: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(e26e3a12-8e37-4b30-aa45-45b26888524c, queue=dev-awx-0) 2021-04-08 15:34:02,285 DEBUG awx.main.dispatch delivered e26e3a12-8e37-4b30-aa45-45b26888524c to worker[192] qsize 0 2021-04-08 15:34:02,596 DEBUG awx.main.dispatch task e26e3a12-8e37-4b30-aa45-45b26888524c starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:34:02,872 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(68fe254c-062a-4c4d-8d70-b523cb32caa5, queue=awx_private_queue) [2021-04-08 15:34:02,872: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(68fe254c-062a-4c4d-8d70-b523cb32caa5, queue=awx_private_queue) 2021-04-08 15:34:03,991 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper for group in InstanceGroup.objects.filter(credential__isnull=False).iterator(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql cursor.close() psycopg2.errors.InvalidCursorName: cursor "_django_curs_140262630049344_4" does not exist 2021-04-08 15:34:04,004 DEBUG awx.main.dispatch task e26e3a12-8e37-4b30-aa45-45b26888524c is finished 2021-04-08 15:34:04,004 DEBUG awx.main.dispatch delivered 68fe254c-062a-4c4d-8d70-b523cb32caa5 to worker[177] qsize 0 2021-04-08 15:34:04,356 DEBUG awx.main.dispatch task 68fe254c-062a-4c4d-8d70-b523cb32caa5 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:34:04,688 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:34:04,718 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:34:04,736 DEBUG awx.main.dispatch task 68fe254c-062a-4c4d-8d70-b523cb32caa5 is finished 2021-04-08 15:34:22,474 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(85f9ba1c-44e4-42c5-8a62-61d366724a0a, queue=awx_private_queue) [2021-04-08 15:34:22,474: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(85f9ba1c-44e4-42c5-8a62-61d366724a0a, queue=awx_private_queue) 2021-04-08 15:34:29,805 DEBUG awx.main.dispatch delivered 85f9ba1c-44e4-42c5-8a62-61d366724a0a to worker[192] qsize 0 2021-04-08 15:34:29,865 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7d2e028d-6d18-4ea7-9795-c7f1c3b4cf64, queue=awx_private_queue) [2021-04-08 15:34:29,865: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7d2e028d-6d18-4ea7-9795-c7f1c3b4cf64, queue=awx_private_queue) 2021-04-08 15:34:29,966 DEBUG awx.main.dispatch delivered 7d2e028d-6d18-4ea7-9795-c7f1c3b4cf64 to worker[177] qsize 0 2021-04-08 15:34:29,861 DEBUG awx.main.dispatch task 85f9ba1c-44e4-42c5-8a62-61d366724a0a starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:34:30,102 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:34:30,107 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:34:30,209 DEBUG awx.main.dispatch task 85f9ba1c-44e4-42c5-8a62-61d366724a0a is finished 2021-04-08 15:34:30,149 DEBUG awx.main.dispatch task 7d2e028d-6d18-4ea7-9795-c7f1c3b4cf64 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:34:32,043 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:34:32,059 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:34:32,065 DEBUG awx.main.dispatch task 7d2e028d-6d18-4ea7-9795-c7f1c3b4cf64 is finished 2021-04-08 15:34:49,812 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(85a96068-8d16-42a5-825f-7fa00a6c603c, queue=awx_private_queue) [2021-04-08 15:34:49,812: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(85a96068-8d16-42a5-825f-7fa00a6c603c, queue=awx_private_queue) 2021-04-08 15:34:50,588 DEBUG awx.main.dispatch delivered 85a96068-8d16-42a5-825f-7fa00a6c603c to worker[195] qsize 0 2021-04-08 15:34:50,614 DEBUG awx.main.dispatch task 85a96068-8d16-42a5-825f-7fa00a6c603c starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:34:54,666 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:34:54,671 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:34:54,678 DEBUG awx.main.dispatch task 85a96068-8d16-42a5-825f-7fa00a6c603c is finished RESULT 2 OKREADY 2021-04-08 15:35:00,583 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(41cf98a4-fc1d-4683-a77f-0ad725a5d259, queue=awx_private_queue) [2021-04-08 15:35:00,583: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(41cf98a4-fc1d-4683-a77f-0ad725a5d259, queue=awx_private_queue) 2021-04-08 15:35:04,456 DEBUG awx.main.dispatch delivered 41cf98a4-fc1d-4683-a77f-0ad725a5d259 to worker[180] qsize 0 2021-04-08 15:35:04,485 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(ad2ff666-f3dd-48dd-8c12-05887b347cda, queue=dev-awx-0) [2021-04-08 15:35:04,485: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(ad2ff666-f3dd-48dd-8c12-05887b347cda, queue=dev-awx-0) 2021-04-08 15:35:04,484 DEBUG awx.main.dispatch task 41cf98a4-fc1d-4683-a77f-0ad725a5d259 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:35:04,584 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:35:04,548 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a45c8e2e-710b-4157-91b7-50a6a103c811, queue=dev-awx-0) [2021-04-08 15:35:04,548: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a45c8e2e-710b-4157-91b7-50a6a103c811, queue=dev-awx-0) 2021-04-08 15:35:04,599 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:35:04,606 DEBUG awx.main.dispatch task 41cf98a4-fc1d-4683-a77f-0ad725a5d259 is finished 2021-04-08 15:35:04,617 ERROR awx.main.dispatch failed to write inbound message Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write self.cleanup() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup reaper.reap(excluded_uuids=running_uuids) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap (changed, me) = Instance.objects.get_or_register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register return self.register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register if instance.exists(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists return self.query.has_results(using=self.db) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results return compiler.has_results() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results return bool(self.execute_sql(SINGLE)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ 2021-04-08 15:35:04,620 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5046baa1-7c67-431f-9c8e-b2eb6a55350f, queue=awx_private_queue) [2021-04-08 15:35:04,620: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5046baa1-7c67-431f-9c8e-b2eb6a55350f, queue=awx_private_queue) 2021-04-08 15:35:04,640 DEBUG awx.main.dispatch delivered a45c8e2e-710b-4157-91b7-50a6a103c811 to worker[195] qsize 0 2021-04-08 15:35:04,657 DEBUG awx.main.dispatch delivered 5046baa1-7c67-431f-9c8e-b2eb6a55350f to worker[177] qsize 0 2021-04-08 15:35:06,720 DEBUG awx.main.dispatch task a45c8e2e-710b-4157-91b7-50a6a103c811 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:35:17,030 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper for group in InstanceGroup.objects.filter(credential__isnull=False).iterator(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql cursor.close() psycopg2.errors.InvalidCursorName: cursor "_django_curs_140262630049344_1" does not exist 2021-04-08 15:35:17,041 DEBUG awx.main.dispatch task a45c8e2e-710b-4157-91b7-50a6a103c811 is finished 2021-04-08 15:35:04,703 DEBUG awx.main.dispatch task 5046baa1-7c67-431f-9c8e-b2eb6a55350f starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:35:18,221 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:35:18,227 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:35:18,273 DEBUG awx.main.dispatch task 5046baa1-7c67-431f-9c8e-b2eb6a55350f is finished 2021-04-08 15:35:33,232 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0bc72dfe-9fac-417a-aab0-afa9db0e2603, queue=awx_private_queue) [2021-04-08 15:35:33,232: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0bc72dfe-9fac-417a-aab0-afa9db0e2603, queue=awx_private_queue) 2021-04-08 15:35:33,475 DEBUG awx.main.dispatch delivered 0bc72dfe-9fac-417a-aab0-afa9db0e2603 to worker[192] qsize 0 2021-04-08 15:35:33,476 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(56e9534c-bb88-42c0-8654-4462ec79af45, queue=awx_private_queue) [2021-04-08 15:35:33,476: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(56e9534c-bb88-42c0-8654-4462ec79af45, queue=awx_private_queue) 2021-04-08 15:35:33,499 DEBUG awx.main.dispatch delivered 56e9534c-bb88-42c0-8654-4462ec79af45 to worker[195] qsize 0 2021-04-08 15:35:33,509 DEBUG awx.main.dispatch task 0bc72dfe-9fac-417a-aab0-afa9db0e2603 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:35:33,582 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:35:33,587 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:35:33,592 DEBUG awx.main.dispatch task 0bc72dfe-9fac-417a-aab0-afa9db0e2603 is finished 2021-04-08 15:35:33,661 DEBUG awx.main.dispatch task 56e9534c-bb88-42c0-8654-4462ec79af45 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:35:35,174 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:35:35,187 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:35:35,200 DEBUG awx.main.dispatch task 56e9534c-bb88-42c0-8654-4462ec79af45 is finished 2021-04-08 15:35:46,050 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(89d24754-8c3a-42d2-b299-863be165a6ea, queue=awx_private_queue) [2021-04-08 15:35:46,050: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(89d24754-8c3a-42d2-b299-863be165a6ea, queue=awx_private_queue) RESULT 2 OKREADY 2021-04-08 15:36:00,479 DEBUG awx.main.dispatch delivered 89d24754-8c3a-42d2-b299-863be165a6ea to worker[192] qsize 0 2021-04-08 15:36:00,747 DEBUG awx.main.dispatch task 89d24754-8c3a-42d2-b299-863be165a6ea starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:36:02,037 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:36:02,038 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:36:02,043 DEBUG awx.main.dispatch task 89d24754-8c3a-42d2-b299-863be165a6ea is finished 2021-04-08 15:36:03,487 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(930b67a6-4602-4668-a1bc-ab17be25091a, queue=awx_private_queue) [2021-04-08 15:36:03,487: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(930b67a6-4602-4668-a1bc-ab17be25091a, queue=awx_private_queue) 2021-04-08 15:36:03,874 DEBUG awx.main.dispatch delivered 930b67a6-4602-4668-a1bc-ab17be25091a to worker[177] qsize 0 2021-04-08 15:36:03,906 DEBUG awx.main.dispatch task 930b67a6-4602-4668-a1bc-ab17be25091a starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:36:03,951 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:36:03,959 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:36:03,969 DEBUG awx.main.dispatch task 930b67a6-4602-4668-a1bc-ab17be25091a is finished 2021-04-08 15:36:04,460 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(78659a94-cc2b-4ba9-be96-8d6a6847d63d, queue=dev-awx-0) [2021-04-08 15:36:04,460: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(78659a94-cc2b-4ba9-be96-8d6a6847d63d, queue=dev-awx-0) 2021-04-08 15:36:06,404 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(41b681f3-3bd5-4daa-af60-fa5721db310e, queue=dev-awx-0) [2021-04-08 15:36:06,404: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(41b681f3-3bd5-4daa-af60-fa5721db310e, queue=dev-awx-0) 2021-04-08 15:36:06,407 ERROR awx.main.dispatch failed to write inbound message Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write self.cleanup() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup reaper.reap(excluded_uuids=running_uuids) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap (changed, me) = Instance.objects.get_or_register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register return self.register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register if instance.exists(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists return self.query.has_results(using=self.db) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results return compiler.has_results() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results return bool(self.execute_sql(SINGLE)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ 2021-04-08 15:36:06,960 DEBUG awx.main.dispatch delivered 41b681f3-3bd5-4daa-af60-fa5721db310e to worker[177] qsize 0 2021-04-08 15:36:06,984 DEBUG awx.main.dispatch task 41b681f3-3bd5-4daa-af60-fa5721db310e starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:36:08,492 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper for group in InstanceGroup.objects.filter(credential__isnull=False).iterator(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql cursor.close() psycopg2.errors.InvalidCursorName: cursor "_django_curs_140262630049344_14" does not exist 2021-04-08 15:36:08,502 DEBUG awx.main.dispatch task 41b681f3-3bd5-4daa-af60-fa5721db310e is finished 2021-04-08 15:36:06,849 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8838a80b-6954-4fae-bf26-b280ab853fc8, queue=awx_private_queue) [2021-04-08 15:36:06,849: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8838a80b-6954-4fae-bf26-b280ab853fc8, queue=awx_private_queue) 2021-04-08 15:36:09,930 DEBUG awx.main.dispatch delivered 8838a80b-6954-4fae-bf26-b280ab853fc8 to worker[180] qsize 0 2021-04-08 15:36:11,379 DEBUG awx.main.dispatch task 8838a80b-6954-4fae-bf26-b280ab853fc8 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:36:32,500 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:36:32,511 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:36:32,521 DEBUG awx.main.dispatch task 8838a80b-6954-4fae-bf26-b280ab853fc8 is finished 2021-04-08 15:36:33,883 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5f660a7f-ba9a-4244-90d9-380ad68120b3, queue=awx_private_queue) [2021-04-08 15:36:33,883: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5f660a7f-ba9a-4244-90d9-380ad68120b3, queue=awx_private_queue) 2021-04-08 15:36:42,078 DEBUG awx.main.dispatch delivered 5f660a7f-ba9a-4244-90d9-380ad68120b3 to worker[192] qsize 0 RESULT 2 OKREADY 2021-04-08 15:36:42,220 DEBUG awx.main.dispatch task 5f660a7f-ba9a-4244-90d9-380ad68120b3 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:37:06,884 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:37:06,897 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:37:06,904 DEBUG awx.main.dispatch task 5f660a7f-ba9a-4244-90d9-380ad68120b3 is finished 2021-04-08 15:36:49,160 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(bf4602ab-38bd-4746-a839-8e2c6b4394ad, queue=awx_private_queue) [2021-04-08 15:36:49,160: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(bf4602ab-38bd-4746-a839-8e2c6b4394ad, queue=awx_private_queue) 2021-04-08 15:37:14,413 DEBUG awx.main.dispatch delivered bf4602ab-38bd-4746-a839-8e2c6b4394ad to worker[177] qsize 0 2021-04-08 15:37:19,964 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3c8e2f34-da61-47e4-a9ef-f0626c227095, queue=awx_private_queue) [2021-04-08 15:37:19,964: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3c8e2f34-da61-47e4-a9ef-f0626c227095, queue=awx_private_queue) 2021-04-08 15:37:47,684 DEBUG awx.main.dispatch delivered 3c8e2f34-da61-47e4-a9ef-f0626c227095 to worker[180] qsize 0 2021-04-08 15:37:39,164 DEBUG awx.main.dispatch task bf4602ab-38bd-4746-a839-8e2c6b4394ad starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:37:48,783 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:37:48,792 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:37:48,821 DEBUG awx.main.dispatch task bf4602ab-38bd-4746-a839-8e2c6b4394ad is finished 2021-04-08 15:37:47,966 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d87fb806-d7fc-4c2c-a72c-14ba08e7a4dd, queue=dev-awx-0) [2021-04-08 15:37:47,966: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d87fb806-d7fc-4c2c-a72c-14ba08e7a4dd, queue=dev-awx-0) 2021-04-08 15:37:48,014 DEBUG awx.main.dispatch task 3c8e2f34-da61-47e4-a9ef-f0626c227095 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:37:54,393 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:37:54,397 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:37:54,403 DEBUG awx.main.dispatch task 3c8e2f34-da61-47e4-a9ef-f0626c227095 is finished 2021-04-08 15:37:53,600 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f145ca89-37a5-41d7-bff7-4731ca731108, queue=dev-awx-0) [2021-04-08 15:37:53,600: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f145ca89-37a5-41d7-bff7-4731ca731108, queue=dev-awx-0) RESULT 2 OKREADY 2021-04-08 15:37:53,603 ERROR awx.main.dispatch failed to write inbound message Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write self.cleanup() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup reaper.reap(excluded_uuids=running_uuids) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap (changed, me) = Instance.objects.get_or_register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register return self.register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register if instance.exists(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists return self.query.has_results(using=self.db) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results return compiler.has_results() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results return bool(self.execute_sql(SINGLE)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ 2021-04-08 15:38:02,437 DEBUG awx.main.dispatch delivered f145ca89-37a5-41d7-bff7-4731ca731108 to worker[180] qsize 0 2021-04-08 15:38:05,731 DEBUG awx.main.dispatch task f145ca89-37a5-41d7-bff7-4731ca731108 starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:38:05,886 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f9565537-ec31-4fb6-9c18-39e4d886154c, queue=awx_private_queue) [2021-04-08 15:38:05,886: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f9565537-ec31-4fb6-9c18-39e4d886154c, queue=awx_private_queue) 2021-04-08 15:38:08,348 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper for group in InstanceGroup.objects.filter(credential__isnull=False).iterator(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql cursor.close() psycopg2.errors.InvalidCursorName: cursor "_django_curs_140262630049344_8" does not exist 2021-04-08 15:38:08,354 DEBUG awx.main.dispatch task f145ca89-37a5-41d7-bff7-4731ca731108 is finished 2021-04-08 15:38:08,385 DEBUG awx.main.dispatch delivered f9565537-ec31-4fb6-9c18-39e4d886154c to worker[180] qsize 0 2021-04-08 15:38:12,152 DEBUG awx.main.dispatch task f9565537-ec31-4fb6-9c18-39e4d886154c starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:38:25,577 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:38:25,583 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:38:25,589 DEBUG awx.main.dispatch task f9565537-ec31-4fb6-9c18-39e4d886154c is finished 2021-04-08 15:38:13,502 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(37e865bb-17b0-487a-aac5-8392dfdd81f5, queue=awx_private_queue) [2021-04-08 15:38:13,502: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(37e865bb-17b0-487a-aac5-8392dfdd81f5, queue=awx_private_queue) 2021-04-08 15:38:25,938 DEBUG awx.main.dispatch delivered 37e865bb-17b0-487a-aac5-8392dfdd81f5 to worker[192] qsize 0 RESULT 2 OKREADY 2021-04-08 15:38:50,873 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(df49ddfb-80e1-4f1d-8ee9-6659c81bf04e, queue=awx_private_queue) [2021-04-08 15:38:50,873: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(df49ddfb-80e1-4f1d-8ee9-6659c81bf04e, queue=awx_private_queue) 2021-04-08 15:39:18,812 DEBUG awx.main.dispatch delivered df49ddfb-80e1-4f1d-8ee9-6659c81bf04e to worker[180] qsize 0 2021-04-08 15:39:19,494 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7de11752-42c8-47f0-9af2-a74855ad1796, queue=awx_private_queue) [2021-04-08 15:39:19,494: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7de11752-42c8-47f0-9af2-a74855ad1796, queue=awx_private_queue) 2021-04-08 15:39:19,857 DEBUG awx.main.dispatch delivered 7de11752-42c8-47f0-9af2-a74855ad1796 to worker[177] qsize 0 2021-04-08 15:38:25,961 DEBUG awx.main.dispatch task 37e865bb-17b0-487a-aac5-8392dfdd81f5 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:39:23,142 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:39:23,336 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:39:23,351 DEBUG awx.main.dispatch task 37e865bb-17b0-487a-aac5-8392dfdd81f5 is finished RESULT 2 OKREADY 2021-04-08 15:39:22,100 DEBUG awx.main.dispatch task df49ddfb-80e1-4f1d-8ee9-6659c81bf04e starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-04-08 15:39:19,917 DEBUG awx.main.dispatch task 7de11752-42c8-47f0-9af2-a74855ad1796 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-04-08 15:40:02,351 DEBUG awx.main.scheduler Running Tower task manager. 2021-04-08 15:39:19,859 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(1ffc9511-c92f-4363-af67-c2d5b10ce1b1, queue=dev-awx-0) [2021-04-08 15:39:19,859: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(1ffc9511-c92f-4363-af67-c2d5b10ce1b1, queue=dev-awx-0) 2021-04-08 15:40:04,216 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager TaskManager().schedule() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__ for rampart_group in InstanceGroup.objects.prefetch_related('instances'): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst... ^ 2021-04-08 15:40:04,229 DEBUG awx.main.dispatch task 7de11752-42c8-47f0-9af2-a74855ad1796 is finished 2021-04-08 15:40:05,388 DEBUG awx.main.tasks Starting periodic scheduler 2021-04-08 15:40:17,301 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{} Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work result = self.run_callable(body) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable return _call(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler state = TowerScheduleState.get_solo() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create return self.get(**kwargs), False File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get num = len(clone) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__ self._fetch_all() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe... ^ 2021-04-08 15:40:18,077 DEBUG awx.main.dispatch task df49ddfb-80e1-4f1d-8ee9-6659c81bf04e is finished 2021-04-08 15:40:06,427 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(79ae2c23-5fbd-45fe-88e3-ae4e904c8d0f, queue=dev-awx-0) [2021-04-08 15:40:06,427: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(79ae2c23-5fbd-45fe-88e3-ae4e904c8d0f, queue=dev-awx-0) 2021-04-08 15:40:03,496 ERROR awx.main.dispatch failed to write inbound message Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write self.cleanup() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup reaper.reap(excluded_uuids=running_uuids) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap (changed, me) = Instance.objects.get_or_register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register return self.register() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register if instance.exists(): File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists return self.query.has_results(using=self.db) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results return compiler.has_results() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results return bool(self.execute_sql(SINGLE)) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "main_instance" does not exist LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"... ^ 2021-04-08 15:40:49,802 DEBUG awx.main.dispatch delivered 79ae2c23-5fbd-45fe-88e3-ae4e904c8d0f to worker[180] qsize 0 RESULT 2 OKREADY 2021-04-08 15:40:49,865 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cb33522e-92ac-4bc4-8b8f-b337c09bab0d, queue=awx_private_queue) [2021-04-08 15:40:49,865: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cb33522e-92ac-4bc4-8b8f-b337c09bab0d, queue=awx_private_queue) RESULT 2 OKREADY 2021-04-08 15:40:49,864 DEBUG awx.main.dispatch task 79ae2c23-5fbd-45fe-88e3-ae4e904c8d0f starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-04-08 15:41:46,621 DEBUG awx.main.dispatch delivered cb33522e-92ac-4bc4-8b8f-b337c09bab0d to worker[195] qsize 0 RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY