By type
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"elapsed": 0,
"match_groupdict": {},
"match_groups": [],
"path": null,
"port": 5432,
"search_regex": null,
"state": "started"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"db": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
}
Default organization added.
Demo Credential, Inventory, and Job Template added.
(changed: True)
2021-05-16 10:45:02,569 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(0df85370-1a96-40ec-9b0e-3efcdad3d929, queue=awx_private_queue)
Successfully registered instance dev-awx-0
(changed: True)
2021-05-16 10:45:05,936 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(a3315a4c-a5ce-49b2-bb49-960f0f12f379, queue=awx_private_queue)
2021-05-16 10:45:05,969 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(61da78b4-7bc6-4946-9c78-0b5d6680efff, queue=awx_private_queue)
Creating instance group tower
(changed: True)
/usr/bin/launch_awx_task.sh: line 27: unset: `#!/bin/sh': not a valid identifier
2021-05-16 10:45:06,615 INFO RPC interface 'supervisor' initialized
2021-05-16 10:45:06,615 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2021-05-16 10:45:06,615 INFO supervisord started with pid 151
2021-05-16 10:45:07,619 INFO spawned: 'awx-config-watcher' with pid 154
2021-05-16 10:45:07,621 INFO spawned: 'dispatcher' with pid 155
2021-05-16 10:45:07,624 INFO spawned: 'callback-receiver' with pid 156
2021-05-16 10:45:07,627 INFO spawned: 'channels-worker' with pid 157
READY
2021-05-16 10:45:08,726 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-16 10:45:08,726 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-16 10:45:08,726 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-16 10:45:08,726 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-16 10:45:10,565 ERROR celery.beat Removing corrupted schedule file '/var/lib/awx/beat.db': error(11, 'Resource temporarily unavailable')
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 485, in setup_schedule
self._store = self._open_schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 475, in _open_schedule
return self.persistence.open(self.schedule_filename, writeback=True)
File "/usr/lib64/python3.6/shelve.py", line 243, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/usr/lib64/python3.6/shelve.py", line 227, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
File "/usr/lib64/python3.6/dbm/__init__.py", line 94, in open
return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable
2021-05-16 10:45:10,583 - INFO - runworker - Using single-threaded worker.
2021-05-16 10:45:10,584 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer)
2021-05-16 10:45:10,578 WARNING awx.main.dispatch scaling up worker pid:177
2021-05-16 10:45:10,584 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive
2021-05-16 10:45:10,606 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2021-05-16 10:45:10,606 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2021-05-16 10:45:10,597 WARNING awx.main.dispatch scaling up worker pid:178
2021-05-16 10:45:10,613 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:180
2021-05-16 10:45:10,613 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:180
2021-05-16 10:45:10,620 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:181
2021-05-16 10:45:10,620 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:181
2021-05-16 10:45:10,628 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:183
2021-05-16 10:45:10,628 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:183
2021-05-16 10:45:10,621 WARNING awx.main.dispatch scaling up worker pid:182
2021-05-16 10:45:10,641 WARNING awx.main.dispatch scaling up worker pid:184
2021-05-16 10:45:10,648 DEBUG awx.main.tasks Syncing Schedules
2021-05-16 10:45:10,863 DEBUG awx.main.tasks Waited 0.0009059906005859375 seconds to obtain lock name: cluster_policy_lock
2021-05-16 10:45:10,873 DEBUG awx.main.tasks Total non-isolated instances:1 available for policy: 1
2021-05-16 10:45:10,873 DEBUG awx.main.tasks Policy percentage, adding Instances [1] to Group tower
2021-05-16 10:45:10,873 DEBUG awx.main.tasks Adding instances [1] to group tower
2021-05-16 10:45:10,879 DEBUG awx.main.tasks Cluster policy computation finished in 0.015296220779418945 seconds
2021-05-16 10:45:10,879 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 10:45:10,904 DEBUG awx.main.dispatch listening on ['awx_private_queue [direct]', 'dev-awx-0 [direct]', 'tower_broadcast_all_dev-awx-0 [fanout]']
2021-05-16 10:45:30,760 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9ab1f9d8-e62d-4687-9a4f-aea0e9fac451, queue=awx_private_queue)
[2021-05-16 10:45:30,760: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9ab1f9d8-e62d-4687-9a4f-aea0e9fac451, queue=awx_private_queue)
2021-05-16 10:45:30,781 DEBUG awx.main.dispatch delivered 9ab1f9d8-e62d-4687-9a4f-aea0e9fac451 to worker[182] qsize 0
2021-05-16 10:45:30,787 DEBUG awx.main.dispatch task 9ab1f9d8-e62d-4687-9a4f-aea0e9fac451 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:45:30,789 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:45:30,817 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:45:30,875 DEBUG awx.main.dispatch task 9ab1f9d8-e62d-4687-9a4f-aea0e9fac451 is finished
2021-05-16 10:45:40,720 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0de79cf3-8d4d-43a1-ad37-7b4c392b5551, queue=awx_private_queue)
[2021-05-16 10:45:40,720: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0de79cf3-8d4d-43a1-ad37-7b4c392b5551, queue=awx_private_queue)
2021-05-16 10:45:40,751 DEBUG awx.main.dispatch delivered 0de79cf3-8d4d-43a1-ad37-7b4c392b5551 to worker[184] qsize 0
2021-05-16 10:45:40,760 DEBUG awx.main.dispatch task 0de79cf3-8d4d-43a1-ad37-7b4c392b5551 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:45:40,782 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:45:40,803 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:45:40.798479+00:00
2021-05-16 10:45:40,830 DEBUG awx.main.dispatch task 0de79cf3-8d4d-43a1-ad37-7b4c392b5551 is finished
2021-05-16 10:45:50,728 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(53c834a5-b8c0-489e-a9ff-f62c48b6edb4, queue=awx_private_queue)
[2021-05-16 10:45:50,728: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(53c834a5-b8c0-489e-a9ff-f62c48b6edb4, queue=awx_private_queue)
2021-05-16 10:45:50,743 DEBUG awx.main.dispatch delivered 53c834a5-b8c0-489e-a9ff-f62c48b6edb4 to worker[184] qsize 0
2021-05-16 10:45:50,749 DEBUG awx.main.dispatch task 53c834a5-b8c0-489e-a9ff-f62c48b6edb4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:45:50,750 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:45:50,778 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:45:50,825 DEBUG awx.main.dispatch task 53c834a5-b8c0-489e-a9ff-f62c48b6edb4 is finished
RESULT 2
OKREADY
2021-05-16 10:46:10,716 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(642cf0db-65b5-4cb0-a922-f38a0a0f9f17, queue=dev-awx-0)
[2021-05-16 10:46:10,716: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(642cf0db-65b5-4cb0-a922-f38a0a0f9f17, queue=dev-awx-0)
2021-05-16 10:46:10,933 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(aad3001d-4762-4559-a685-1bdc2789ae2a, queue=dev-awx-0)
[2021-05-16 10:46:10,933: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(aad3001d-4762-4559-a685-1bdc2789ae2a, queue=dev-awx-0)
2021-05-16 10:46:10,948 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8b927bdf-1f9f-4a36-b0c7-3531bf88c986, queue=awx_private_queue)
[2021-05-16 10:46:10,948: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8b927bdf-1f9f-4a36-b0c7-3531bf88c986, queue=awx_private_queue)
2021-05-16 10:46:10,964 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9cace94e-6c06-4127-9ccd-9dbea74bf858, queue=awx_private_queue)
[2021-05-16 10:46:10,964: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9cace94e-6c06-4127-9ccd-9dbea74bf858, queue=awx_private_queue)
2021-05-16 10:46:10,965 DEBUG awx.main.dispatch delivered 642cf0db-65b5-4cb0-a922-f38a0a0f9f17 to worker[182] qsize 0
2021-05-16 10:46:10,969 DEBUG awx.main.dispatch delivered aad3001d-4762-4559-a685-1bdc2789ae2a to worker[177] qsize 0
2021-05-16 10:46:10,977 DEBUG awx.main.dispatch delivered 8b927bdf-1f9f-4a36-b0c7-3531bf88c986 to worker[184] qsize 0
2021-05-16 10:46:10,971 DEBUG awx.main.dispatch task 642cf0db-65b5-4cb0-a922-f38a0a0f9f17 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 10:46:10,979 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 10:46:10,980 DEBUG awx.main.dispatch task 8b927bdf-1f9f-4a36-b0c7-3531bf88c986 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:46:10,983 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:46:10,980 DEBUG awx.main.dispatch task aad3001d-4762-4559-a685-1bdc2789ae2a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 10:46:10,997 DEBUG awx.main.dispatch delivered 9cace94e-6c06-4127-9ccd-9dbea74bf858 to worker[178] qsize 0
2021-05-16 10:46:11,000 DEBUG awx.main.dispatch task 9cace94e-6c06-4127-9ccd-9dbea74bf858 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:46:11,022 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:46:11,026 DEBUG awx.main.dispatch task aad3001d-4762-4559-a685-1bdc2789ae2a is finished
2021-05-16 10:46:11,028 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:46:11,036 DEBUG awx.main.dispatch task 642cf0db-65b5-4cb0-a922-f38a0a0f9f17 is finished
2021-05-16 10:46:11,034 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:45:40.782536+00:00
2021-05-16 10:46:11,062 DEBUG awx.main.dispatch task 9cace94e-6c06-4127-9ccd-9dbea74bf858 is finished
2021-05-16 10:46:11,071 DEBUG awx.main.dispatch task 8b927bdf-1f9f-4a36-b0c7-3531bf88c986 is finished
2021-05-16 10:46:30,965 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d1334b2e-970d-4d3b-bb5c-cfed37b4d26d, queue=awx_private_queue)
[2021-05-16 10:46:30,965: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d1334b2e-970d-4d3b-bb5c-cfed37b4d26d, queue=awx_private_queue)
2021-05-16 10:46:30,989 DEBUG awx.main.dispatch delivered d1334b2e-970d-4d3b-bb5c-cfed37b4d26d to worker[184] qsize 0
2021-05-16 10:46:30,995 DEBUG awx.main.dispatch task d1334b2e-970d-4d3b-bb5c-cfed37b4d26d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:46:30,998 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:46:31,023 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:46:31,064 DEBUG awx.main.dispatch task d1334b2e-970d-4d3b-bb5c-cfed37b4d26d is finished
2021-05-16 10:46:40,964 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a4c7dc5c-b632-4ca9-9086-a165df94f115, queue=awx_private_queue)
[2021-05-16 10:46:40,964: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a4c7dc5c-b632-4ca9-9086-a165df94f115, queue=awx_private_queue)
2021-05-16 10:46:40,980 DEBUG awx.main.dispatch delivered a4c7dc5c-b632-4ca9-9086-a165df94f115 to worker[182] qsize 0
2021-05-16 10:46:40,985 DEBUG awx.main.dispatch task a4c7dc5c-b632-4ca9-9086-a165df94f115 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:46:41,003 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:46:41,008 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:46:11.022967+00:00
2021-05-16 10:46:41,041 DEBUG awx.main.dispatch task a4c7dc5c-b632-4ca9-9086-a165df94f115 is finished
2021-05-16 10:46:50,958 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d4ab0c24-edf7-450e-9314-4fd5edd16784, queue=awx_private_queue)
[2021-05-16 10:46:50,958: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d4ab0c24-edf7-450e-9314-4fd5edd16784, queue=awx_private_queue)
2021-05-16 10:46:50,972 DEBUG awx.main.dispatch delivered d4ab0c24-edf7-450e-9314-4fd5edd16784 to worker[178] qsize 0
2021-05-16 10:46:50,977 DEBUG awx.main.dispatch task d4ab0c24-edf7-450e-9314-4fd5edd16784 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:46:50,979 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:46:51,144 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:46:51,228 DEBUG awx.main.dispatch task d4ab0c24-edf7-450e-9314-4fd5edd16784 is finished
RESULT 2
OKREADY
2021-05-16 10:47:10,724 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(98760464-dc87-4ebe-b4e1-7e94957dd8fb, queue=dev-awx-0)
[2021-05-16 10:47:10,724: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(98760464-dc87-4ebe-b4e1-7e94957dd8fb, queue=dev-awx-0)
2021-05-16 10:47:10,794 DEBUG awx.main.dispatch delivered 98760464-dc87-4ebe-b4e1-7e94957dd8fb to worker[177] qsize 0
2021-05-16 10:47:10,806 DEBUG awx.main.dispatch task 98760464-dc87-4ebe-b4e1-7e94957dd8fb starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 10:47:10,811 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 10:47:10,858 DEBUG awx.main.dispatch task 98760464-dc87-4ebe-b4e1-7e94957dd8fb is finished
2021-05-16 10:47:10,933 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b32ba555-39a0-4589-a700-0abbc6ed0ed4, queue=dev-awx-0)
[2021-05-16 10:47:10,933: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b32ba555-39a0-4589-a700-0abbc6ed0ed4, queue=dev-awx-0)
2021-05-16 10:47:10,946 DEBUG awx.main.dispatch delivered b32ba555-39a0-4589-a700-0abbc6ed0ed4 to worker[177] qsize 0
2021-05-16 10:47:10,948 DEBUG awx.main.dispatch task b32ba555-39a0-4589-a700-0abbc6ed0ed4 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 10:47:10,958 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c45f19bc-b57b-44f5-90ec-af59bad612a8, queue=awx_private_queue)
[2021-05-16 10:47:10,958: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c45f19bc-b57b-44f5-90ec-af59bad612a8, queue=awx_private_queue)
2021-05-16 10:47:10,965 DEBUG awx.main.dispatch task b32ba555-39a0-4589-a700-0abbc6ed0ed4 is finished
2021-05-16 10:47:10,973 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a95a9a8d-94bd-41e5-81da-868d25eb8e2f, queue=awx_private_queue)
[2021-05-16 10:47:10,973: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a95a9a8d-94bd-41e5-81da-868d25eb8e2f, queue=awx_private_queue)
2021-05-16 10:47:10,972 DEBUG awx.main.dispatch delivered c45f19bc-b57b-44f5-90ec-af59bad612a8 to worker[182] qsize 0
2021-05-16 10:47:11,147 DEBUG awx.main.dispatch task c45f19bc-b57b-44f5-90ec-af59bad612a8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:47:11,149 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:47:11,148 DEBUG awx.main.dispatch delivered a95a9a8d-94bd-41e5-81da-868d25eb8e2f to worker[184] qsize 0
2021-05-16 10:47:11,151 DEBUG awx.main.dispatch task a95a9a8d-94bd-41e5-81da-868d25eb8e2f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:47:11,173 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:47:11,179 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:46:41.003832+00:00
2021-05-16 10:47:11,191 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:47:11,215 DEBUG awx.main.dispatch task a95a9a8d-94bd-41e5-81da-868d25eb8e2f is finished
2021-05-16 10:47:11,244 DEBUG awx.main.dispatch task c45f19bc-b57b-44f5-90ec-af59bad612a8 is finished
2021-05-16 10:47:30,967 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c6b86e5a-33ca-45c4-bd6a-26ffcc18baf6, queue=awx_private_queue)
[2021-05-16 10:47:30,967: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c6b86e5a-33ca-45c4-bd6a-26ffcc18baf6, queue=awx_private_queue)
2021-05-16 10:47:30,995 DEBUG awx.main.dispatch delivered c6b86e5a-33ca-45c4-bd6a-26ffcc18baf6 to worker[177] qsize 0
2021-05-16 10:47:31,002 DEBUG awx.main.dispatch task c6b86e5a-33ca-45c4-bd6a-26ffcc18baf6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:47:31,006 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:47:31,031 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:47:31,091 DEBUG awx.main.dispatch task c6b86e5a-33ca-45c4-bd6a-26ffcc18baf6 is finished
2021-05-16 10:47:40,975 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(223cc84b-4a5a-439b-9737-1b1f06b5d9c7, queue=awx_private_queue)
[2021-05-16 10:47:40,975: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(223cc84b-4a5a-439b-9737-1b1f06b5d9c7, queue=awx_private_queue)
2021-05-16 10:47:40,991 DEBUG awx.main.dispatch delivered 223cc84b-4a5a-439b-9737-1b1f06b5d9c7 to worker[177] qsize 0
2021-05-16 10:47:40,994 DEBUG awx.main.dispatch task 223cc84b-4a5a-439b-9737-1b1f06b5d9c7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:47:41,096 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:47:41,102 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:47:11.173694+00:00
2021-05-16 10:47:41,118 DEBUG awx.main.dispatch task 223cc84b-4a5a-439b-9737-1b1f06b5d9c7 is finished
2021-05-16 10:47:50,963 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(760b5b6f-0c89-4d93-96bd-6c1a555edc07, queue=awx_private_queue)
[2021-05-16 10:47:50,963: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(760b5b6f-0c89-4d93-96bd-6c1a555edc07, queue=awx_private_queue)
2021-05-16 10:47:50,990 DEBUG awx.main.dispatch delivered 760b5b6f-0c89-4d93-96bd-6c1a555edc07 to worker[178] qsize 0
2021-05-16 10:47:50,994 DEBUG awx.main.dispatch task 760b5b6f-0c89-4d93-96bd-6c1a555edc07 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:47:50,998 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:47:51,024 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:47:51,069 DEBUG awx.main.dispatch task 760b5b6f-0c89-4d93-96bd-6c1a555edc07 is finished
RESULT 2
OKREADY
2021-05-16 10:48:10,725 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(aaeb63f3-ecee-400e-aea3-0575eb8357d4, queue=dev-awx-0)
[2021-05-16 10:48:10,725: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(aaeb63f3-ecee-400e-aea3-0575eb8357d4, queue=dev-awx-0)
2021-05-16 10:48:10,790 DEBUG awx.main.dispatch delivered aaeb63f3-ecee-400e-aea3-0575eb8357d4 to worker[178] qsize 0
2021-05-16 10:48:10,792 DEBUG awx.main.dispatch task aaeb63f3-ecee-400e-aea3-0575eb8357d4 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 10:48:10,793 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 10:48:10,888 DEBUG awx.main.dispatch task aaeb63f3-ecee-400e-aea3-0575eb8357d4 is finished
2021-05-16 10:48:10,933 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(314a56d8-245b-425a-8c9b-655762c4acc5, queue=dev-awx-0)
[2021-05-16 10:48:10,933: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(314a56d8-245b-425a-8c9b-655762c4acc5, queue=dev-awx-0)
2021-05-16 10:48:10,947 DEBUG awx.main.dispatch delivered 314a56d8-245b-425a-8c9b-655762c4acc5 to worker[177] qsize 0
2021-05-16 10:48:10,949 DEBUG awx.main.dispatch task 314a56d8-245b-425a-8c9b-655762c4acc5 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 10:48:10,963 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e63013f0-86d5-4254-9f67-3008a057f754, queue=awx_private_queue)
[2021-05-16 10:48:10,963: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e63013f0-86d5-4254-9f67-3008a057f754, queue=awx_private_queue)
2021-05-16 10:48:10,978 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(adba9faa-74e0-4edf-917d-556137cdd1ff, queue=awx_private_queue)
[2021-05-16 10:48:10,978: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(adba9faa-74e0-4edf-917d-556137cdd1ff, queue=awx_private_queue)
2021-05-16 10:48:11,002 DEBUG awx.main.dispatch task 314a56d8-245b-425a-8c9b-655762c4acc5 is finished
2021-05-16 10:48:10,977 DEBUG awx.main.dispatch delivered e63013f0-86d5-4254-9f67-3008a057f754 to worker[178] qsize 0
2021-05-16 10:48:11,145 DEBUG awx.main.dispatch delivered adba9faa-74e0-4edf-917d-556137cdd1ff to worker[182] qsize 0
2021-05-16 10:48:11,145 DEBUG awx.main.dispatch task e63013f0-86d5-4254-9f67-3008a057f754 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:48:11,147 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:48:11,149 DEBUG awx.main.dispatch task adba9faa-74e0-4edf-917d-556137cdd1ff starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:48:11,174 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:48:11,181 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:47:41.097089+00:00
2021-05-16 10:48:11,196 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:48:11,220 DEBUG awx.main.dispatch task adba9faa-74e0-4edf-917d-556137cdd1ff is finished
2021-05-16 10:48:11,265 DEBUG awx.main.dispatch task e63013f0-86d5-4254-9f67-3008a057f754 is finished
2021-05-16 10:48:30,985 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(649c4255-5404-4bb8-96fb-82b2c7df744c, queue=awx_private_queue)
[2021-05-16 10:48:30,985: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(649c4255-5404-4bb8-96fb-82b2c7df744c, queue=awx_private_queue)
2021-05-16 10:48:31,008 DEBUG awx.main.dispatch delivered 649c4255-5404-4bb8-96fb-82b2c7df744c to worker[177] qsize 0
2021-05-16 10:48:31,014 DEBUG awx.main.dispatch task 649c4255-5404-4bb8-96fb-82b2c7df744c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:48:31,018 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:48:31,208 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:48:31,542 DEBUG awx.main.dispatch task 649c4255-5404-4bb8-96fb-82b2c7df744c is finished
2021-05-16 10:48:40,979 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9f202d64-a1ea-4ba1-8093-6eb2ff8b53f8, queue=awx_private_queue)
[2021-05-16 10:48:40,979: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9f202d64-a1ea-4ba1-8093-6eb2ff8b53f8, queue=awx_private_queue)
2021-05-16 10:48:41,002 DEBUG awx.main.dispatch delivered 9f202d64-a1ea-4ba1-8093-6eb2ff8b53f8 to worker[177] qsize 0
2021-05-16 10:48:41,005 DEBUG awx.main.dispatch task 9f202d64-a1ea-4ba1-8093-6eb2ff8b53f8 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:48:41,387 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:48:41,393 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:48:11.174187+00:00
2021-05-16 10:48:41,511 DEBUG awx.main.dispatch task 9f202d64-a1ea-4ba1-8093-6eb2ff8b53f8 is finished
2021-05-16 10:48:50,983 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5874ee67-b579-4557-8454-66d419c5fe84, queue=awx_private_queue)
[2021-05-16 10:48:50,983: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5874ee67-b579-4557-8454-66d419c5fe84, queue=awx_private_queue)
2021-05-16 10:48:50,999 DEBUG awx.main.dispatch delivered 5874ee67-b579-4557-8454-66d419c5fe84 to worker[182] qsize 0
2021-05-16 10:48:51,002 DEBUG awx.main.dispatch task 5874ee67-b579-4557-8454-66d419c5fe84 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:48:51,005 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:48:51,023 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:48:51,082 DEBUG awx.main.dispatch task 5874ee67-b579-4557-8454-66d419c5fe84 is finished
RESULT 2
OKREADY
2021-05-16 10:49:10,736 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(0afb8902-5c65-48aa-be81-347350035c13, queue=dev-awx-0)
[2021-05-16 10:49:10,736: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(0afb8902-5c65-48aa-be81-347350035c13, queue=dev-awx-0)
2021-05-16 10:49:10,807 DEBUG awx.main.dispatch delivered 0afb8902-5c65-48aa-be81-347350035c13 to worker[178] qsize 0
2021-05-16 10:49:10,812 DEBUG awx.main.dispatch task 0afb8902-5c65-48aa-be81-347350035c13 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 10:49:10,816 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 10:49:10,933 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b1f1844d-67d5-4047-8559-27bdabd9f2d8, queue=dev-awx-0)
[2021-05-16 10:49:10,933: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b1f1844d-67d5-4047-8559-27bdabd9f2d8, queue=dev-awx-0)
2021-05-16 10:49:11,346 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f8965a38-ee07-4d63-ba1b-ca61c4d1c665, queue=awx_private_queue)
[2021-05-16 10:49:11,346: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f8965a38-ee07-4d63-ba1b-ca61c4d1c665, queue=awx_private_queue)
2021-05-16 10:49:11,365 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(eb335dd3-463c-40af-a91a-5b488cfe4708, queue=awx_private_queue)
[2021-05-16 10:49:11,365: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(eb335dd3-463c-40af-a91a-5b488cfe4708, queue=awx_private_queue)
2021-05-16 10:49:11,345 DEBUG awx.main.dispatch delivered b1f1844d-67d5-4047-8559-27bdabd9f2d8 to worker[177] qsize 0
2021-05-16 10:49:11,523 DEBUG awx.main.dispatch delivered f8965a38-ee07-4d63-ba1b-ca61c4d1c665 to worker[182] qsize 0
2021-05-16 10:49:11,524 DEBUG awx.main.dispatch delivered eb335dd3-463c-40af-a91a-5b488cfe4708 to worker[184] qsize 0
2021-05-16 10:49:11,525 DEBUG awx.main.dispatch task b1f1844d-67d5-4047-8559-27bdabd9f2d8 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 10:49:11,528 DEBUG awx.main.dispatch task eb335dd3-463c-40af-a91a-5b488cfe4708 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:49:11,532 DEBUG awx.main.dispatch task f8965a38-ee07-4d63-ba1b-ca61c4d1c665 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:49:11,536 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:49:11,644 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:49:11,656 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:48:41.387634+00:00
2021-05-16 10:49:11,664 DEBUG awx.main.dispatch task b1f1844d-67d5-4047-8559-27bdabd9f2d8 is finished
2021-05-16 10:49:11,674 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:49:11,746 DEBUG awx.main.dispatch task eb335dd3-463c-40af-a91a-5b488cfe4708 is finished
2021-05-16 10:49:11,748 DEBUG awx.main.dispatch task 0afb8902-5c65-48aa-be81-347350035c13 is finished
2021-05-16 10:49:11,771 DEBUG awx.main.dispatch task f8965a38-ee07-4d63-ba1b-ca61c4d1c665 is finished
2021-05-16 10:49:31,356 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5278bb74-0222-4222-844c-322595d9477e, queue=awx_private_queue)
[2021-05-16 10:49:31,356: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5278bb74-0222-4222-844c-322595d9477e, queue=awx_private_queue)
2021-05-16 10:49:31,368 DEBUG awx.main.dispatch delivered 5278bb74-0222-4222-844c-322595d9477e to worker[178] qsize 0
2021-05-16 10:49:31,370 DEBUG awx.main.dispatch task 5278bb74-0222-4222-844c-322595d9477e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:49:31,371 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:49:31,482 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:49:31,539 DEBUG awx.main.dispatch task 5278bb74-0222-4222-844c-322595d9477e is finished
2021-05-16 10:49:41,365 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a96c3385-7f0b-43c1-ab80-72c26f775c7d, queue=awx_private_queue)
[2021-05-16 10:49:41,365: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a96c3385-7f0b-43c1-ab80-72c26f775c7d, queue=awx_private_queue)
2021-05-16 10:49:41,379 DEBUG awx.main.dispatch delivered a96c3385-7f0b-43c1-ab80-72c26f775c7d to worker[177] qsize 0
2021-05-16 10:49:41,381 DEBUG awx.main.dispatch task a96c3385-7f0b-43c1-ab80-72c26f775c7d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:49:41,398 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:49:41,401 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:49:11.645239+00:00
2021-05-16 10:49:41,437 DEBUG awx.main.dispatch task a96c3385-7f0b-43c1-ab80-72c26f775c7d is finished
2021-05-16 10:49:51,355 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c1b53106-99bf-49ed-93e0-17e0eab2ba47, queue=awx_private_queue)
[2021-05-16 10:49:51,355: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c1b53106-99bf-49ed-93e0-17e0eab2ba47, queue=awx_private_queue)
2021-05-16 10:49:51,368 DEBUG awx.main.dispatch delivered c1b53106-99bf-49ed-93e0-17e0eab2ba47 to worker[177] qsize 0
2021-05-16 10:49:51,370 DEBUG awx.main.dispatch task c1b53106-99bf-49ed-93e0-17e0eab2ba47 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:49:51,372 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:49:51,928 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:49:52,109 DEBUG awx.main.dispatch task c1b53106-99bf-49ed-93e0-17e0eab2ba47 is finished
RESULT 2
OKREADY
2021-05-16 10:50:10,742 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e8511911-0ee4-4383-aeff-0990c24fd4bc, queue=dev-awx-0)
[2021-05-16 10:50:10,742: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e8511911-0ee4-4383-aeff-0990c24fd4bc, queue=dev-awx-0)
2021-05-16 10:50:10,933 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(62454688-df99-4c69-bec5-06c2a06d3f0f, queue=dev-awx-0)
[2021-05-16 10:50:10,933: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(62454688-df99-4c69-bec5-06c2a06d3f0f, queue=dev-awx-0)
2021-05-16 10:50:11,355 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7590342b-22cf-44b4-80cc-e8d4d406c653, queue=awx_private_queue)
[2021-05-16 10:50:11,355: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7590342b-22cf-44b4-80cc-e8d4d406c653, queue=awx_private_queue)
2021-05-16 10:50:12,086 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(67e5c166-ec2a-4b5a-bdb4-ee5373f8caf7, queue=awx_private_queue)
[2021-05-16 10:50:12,086: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(67e5c166-ec2a-4b5a-bdb4-ee5373f8caf7, queue=awx_private_queue)
2021-05-16 10:50:12,692 DEBUG awx.main.dispatch delivered e8511911-0ee4-4383-aeff-0990c24fd4bc to worker[184] qsize 0
2021-05-16 10:50:12,861 DEBUG awx.main.dispatch delivered 62454688-df99-4c69-bec5-06c2a06d3f0f to worker[178] qsize 0
2021-05-16 10:50:12,862 DEBUG awx.main.dispatch delivered 7590342b-22cf-44b4-80cc-e8d4d406c653 to worker[182] qsize 0
2021-05-16 10:50:12,864 DEBUG awx.main.dispatch delivered 67e5c166-ec2a-4b5a-bdb4-ee5373f8caf7 to worker[177] qsize 0
2021-05-16 10:50:12,862 DEBUG awx.main.dispatch task e8511911-0ee4-4383-aeff-0990c24fd4bc starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 10:50:12,866 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 10:50:12,865 DEBUG awx.main.dispatch task 62454688-df99-4c69-bec5-06c2a06d3f0f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 10:50:12,869 DEBUG awx.main.dispatch task 7590342b-22cf-44b4-80cc-e8d4d406c653 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:50:12,875 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:50:12,873 DEBUG awx.main.dispatch task 67e5c166-ec2a-4b5a-bdb4-ee5373f8caf7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:50:13,202 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:50:13,210 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:49:41.398859+00:00
2021-05-16 10:50:13,308 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:50:13,356 DEBUG awx.main.dispatch task 67e5c166-ec2a-4b5a-bdb4-ee5373f8caf7 is finished
2021-05-16 10:50:13,379 DEBUG awx.main.dispatch task 62454688-df99-4c69-bec5-06c2a06d3f0f is finished
2021-05-16 10:50:13,679 DEBUG awx.main.dispatch task e8511911-0ee4-4383-aeff-0990c24fd4bc is finished
2021-05-16 10:50:13,748 DEBUG awx.main.dispatch task 7590342b-22cf-44b4-80cc-e8d4d406c653 is finished
2021-05-16 10:50:31,365 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ad31ff0e-fec5-43e7-8732-270908021803, queue=awx_private_queue)
[2021-05-16 10:50:31,365: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ad31ff0e-fec5-43e7-8732-270908021803, queue=awx_private_queue)
2021-05-16 10:50:31,382 DEBUG awx.main.dispatch delivered ad31ff0e-fec5-43e7-8732-270908021803 to worker[182] qsize 0
2021-05-16 10:50:31,384 DEBUG awx.main.dispatch task ad31ff0e-fec5-43e7-8732-270908021803 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:50:31,387 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:50:32,402 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:50:42,087 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f88d6026-9c83-4689-b351-5a35c0e58819, queue=awx_private_queue)
[2021-05-16 10:50:42,087: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f88d6026-9c83-4689-b351-5a35c0e58819, queue=awx_private_queue)
2021-05-16 10:50:42,100 DEBUG awx.main.dispatch delivered f88d6026-9c83-4689-b351-5a35c0e58819 to worker[177] qsize 0
2021-05-16 10:50:42,102 DEBUG awx.main.dispatch task f88d6026-9c83-4689-b351-5a35c0e58819 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:50:51,366 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5bffa7d0-7397-443f-a9a1-4ab0759ef2b5, queue=awx_private_queue)
[2021-05-16 10:50:51,366: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5bffa7d0-7397-443f-a9a1-4ab0759ef2b5, queue=awx_private_queue)
2021-05-16 10:50:51,383 DEBUG awx.main.dispatch delivered 5bffa7d0-7397-443f-a9a1-4ab0759ef2b5 to worker[178] qsize 0
2021-05-16 10:50:51,385 DEBUG awx.main.dispatch task 5bffa7d0-7397-443f-a9a1-4ab0759ef2b5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:50:51,386 DEBUG awx.main.scheduler Running Tower task manager.
RESULT 2
OKREADY
2021-05-16 10:51:10,746 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f7da1988-2595-4651-b82c-a3e3b01d427e, queue=dev-awx-0)
[2021-05-16 10:51:10,746: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f7da1988-2595-4651-b82c-a3e3b01d427e, queue=dev-awx-0)
2021-05-16 10:51:10,933 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(22785393-004c-4155-a470-2b73cba3b14d, queue=dev-awx-0)
[2021-05-16 10:51:10,933: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(22785393-004c-4155-a470-2b73cba3b14d, queue=dev-awx-0)
2021-05-16 10:51:11,366 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6cf38352-103f-4283-b647-bfa4fd584b24, queue=awx_private_queue)
[2021-05-16 10:51:11,366: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6cf38352-103f-4283-b647-bfa4fd584b24, queue=awx_private_queue)
2021-05-16 10:51:11,685 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:51:12,821 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:50:13.202704+00:00
2021-05-16 10:51:12,838 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-16 10:51:12,838 DEBUG awx.main.dispatch task 5bffa7d0-7397-443f-a9a1-4ab0759ef2b5 is finished
2021-05-16 10:51:12,715 DEBUG awx.main.dispatch delivered f7da1988-2595-4651-b82c-a3e3b01d427e to worker[184] qsize 0
2021-05-16 10:51:12,871 DEBUG awx.main.dispatch delivered 22785393-004c-4155-a470-2b73cba3b14d to worker[178] qsize 0
2021-05-16 10:51:12,870 DEBUG awx.main.dispatch task f7da1988-2595-4651-b82c-a3e3b01d427e starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 10:51:12,872 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 10:51:12,872 DEBUG awx.main.dispatch task 22785393-004c-4155-a470-2b73cba3b14d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 10:51:12,886 WARNING awx.main.dispatch scaling up worker pid:193
2021-05-16 10:51:12,891 DEBUG awx.main.dispatch delivered 6cf38352-103f-4283-b647-bfa4fd584b24 to worker[193] qsize 0
2021-05-16 10:51:12,895 DEBUG awx.main.dispatch task 6cf38352-103f-4283-b647-bfa4fd584b24 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:51:12,897 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:51:12,924 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-16 10:51:12,924 DEBUG awx.main.dispatch task 6cf38352-103f-4283-b647-bfa4fd584b24 is finished
2021-05-16 10:51:13,632 DEBUG awx.main.dispatch task 22785393-004c-4155-a470-2b73cba3b14d is finished
2021-05-16 10:51:12,087 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e572d8df-2592-4899-9712-8ae559d0a78e, queue=awx_private_queue)
[2021-05-16 10:51:12,087: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e572d8df-2592-4899-9712-8ae559d0a78e, queue=awx_private_queue)
2021-05-16 10:51:13,828 DEBUG awx.main.dispatch delivered e572d8df-2592-4899-9712-8ae559d0a78e to worker[193] qsize 0
2021-05-16 10:51:13,830 DEBUG awx.main.dispatch task e572d8df-2592-4899-9712-8ae559d0a78e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:51:13,860 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-05-16 10:51:13,860 DEBUG awx.main.dispatch task e572d8df-2592-4899-9712-8ae559d0a78e is finished
2021-05-16 10:51:14,541 DEBUG awx.main.dispatch task f88d6026-9c83-4689-b351-5a35c0e58819 is finished
2021-05-16 10:51:14,954 DEBUG awx.main.dispatch task f7da1988-2595-4651-b82c-a3e3b01d427e is finished
2021-05-16 10:51:23,829 DEBUG awx.main.dispatch task ad31ff0e-fec5-43e7-8732-270908021803 is finished
2021-05-16 10:51:31,366 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(acaedca1-1c8c-45ae-b75f-195b43792e16, queue=awx_private_queue)
[2021-05-16 10:51:31,366: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(acaedca1-1c8c-45ae-b75f-195b43792e16, queue=awx_private_queue)
2021-05-16 10:51:31,380 DEBUG awx.main.dispatch delivered acaedca1-1c8c-45ae-b75f-195b43792e16 to worker[177] qsize 0
2021-05-16 10:51:31,382 DEBUG awx.main.dispatch task acaedca1-1c8c-45ae-b75f-195b43792e16 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:51:31,383 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:51:38,605 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:51:39,002 DEBUG awx.main.dispatch task acaedca1-1c8c-45ae-b75f-195b43792e16 is finished
2021-05-16 10:51:42,093 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(71c13a25-8884-4c3b-be93-6c29bff2a7bd, queue=awx_private_queue)
[2021-05-16 10:51:42,093: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(71c13a25-8884-4c3b-be93-6c29bff2a7bd, queue=awx_private_queue)
2021-05-16 10:51:42,115 DEBUG awx.main.dispatch delivered 71c13a25-8884-4c3b-be93-6c29bff2a7bd to worker[177] qsize 0
2021-05-16 10:51:42,121 DEBUG awx.main.dispatch task 71c13a25-8884-4c3b-be93-6c29bff2a7bd starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:51:51,366 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9132d7f6-b3b3-4f26-8c2d-de863d99bcf1, queue=awx_private_queue)
[2021-05-16 10:51:51,366: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9132d7f6-b3b3-4f26-8c2d-de863d99bcf1, queue=awx_private_queue)
2021-05-16 10:51:51,380 DEBUG awx.main.dispatch delivered 9132d7f6-b3b3-4f26-8c2d-de863d99bcf1 to worker[182] qsize 0
2021-05-16 10:51:51,385 DEBUG awx.main.dispatch task 9132d7f6-b3b3-4f26-8c2d-de863d99bcf1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:51:51,388 DEBUG awx.main.scheduler Running Tower task manager.
RESULT 2
OKREADY
2021-05-16 10:52:10,756 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8e3cb1f1-3773-4026-a55b-48eb43f17f47, queue=dev-awx-0)
[2021-05-16 10:52:10,756: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8e3cb1f1-3773-4026-a55b-48eb43f17f47, queue=dev-awx-0)
2021-05-16 10:52:10,773 WARNING awx.main.dispatch scaling down worker pid:178
2021-05-16 10:52:10,778 WARNING awx.main.dispatch worker exiting gracefully pid:178
2021-05-16 10:52:10,934 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(6b229a38-f6d5-449e-a529-677e7cafcbc0, queue=dev-awx-0)
[2021-05-16 10:52:10,934: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(6b229a38-f6d5-449e-a529-677e7cafcbc0, queue=dev-awx-0)
2021-05-16 10:52:11,366 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7e799bf9-7350-41f3-86cd-4895df2395fa, queue=awx_private_queue)
[2021-05-16 10:52:11,366: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7e799bf9-7350-41f3-86cd-4895df2395fa, queue=awx_private_queue)
2021-05-16 10:52:12,088 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9b7787cd-1007-4cee-be61-5b9883350126, queue=awx_private_queue)
[2021-05-16 10:52:12,088: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9b7787cd-1007-4cee-be61-5b9883350126, queue=awx_private_queue)
2021-05-16 10:52:22,985 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:52:23,422 DEBUG awx.main.dispatch delivered 8e3cb1f1-3773-4026-a55b-48eb43f17f47 to worker[193] qsize 0
2021-05-16 10:52:23,596 DEBUG awx.main.dispatch delivered 6b229a38-f6d5-449e-a529-677e7cafcbc0 to worker[184] qsize 0
2021-05-16 10:52:23,597 DEBUG awx.main.dispatch task 8e3cb1f1-3773-4026-a55b-48eb43f17f47 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 10:52:23,606 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 10:52:23,605 DEBUG awx.main.dispatch task 6b229a38-f6d5-449e-a529-677e7cafcbc0 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 10:52:23,610 WARNING awx.main.dispatch scaling up worker pid:196
2021-05-16 10:52:23,614 DEBUG awx.main.dispatch delivered 7e799bf9-7350-41f3-86cd-4895df2395fa to worker[196] qsize 0
2021-05-16 10:52:23,618 DEBUG awx.main.dispatch task 7e799bf9-7350-41f3-86cd-4895df2395fa starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:52:23,619 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:52:23,621 WARNING awx.main.dispatch scaling up worker pid:198
2021-05-16 10:52:23,625 DEBUG awx.main.dispatch delivered 9b7787cd-1007-4cee-be61-5b9883350126 to worker[198] qsize 0
2021-05-16 10:52:23,628 DEBUG awx.main.dispatch task 9b7787cd-1007-4cee-be61-5b9883350126 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:52:23,880 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:51:11.685887+00:00
2021-05-16 10:52:23,960 DEBUG awx.main.tasks Not running periodic scheduler, another task holds lock
2021-05-16 10:52:23,960 DEBUG awx.main.dispatch task 9b7787cd-1007-4cee-be61-5b9883350126 is finished
2021-05-16 10:52:23,971 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:52:23,998 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-16 10:52:23,998 DEBUG awx.main.dispatch task 9132d7f6-b3b3-4f26-8c2d-de863d99bcf1 is finished
2021-05-16 10:52:24,312 DEBUG awx.main.dispatch task 71c13a25-8884-4c3b-be93-6c29bff2a7bd is finished
2021-05-16 10:52:24,668 DEBUG awx.main.dispatch task 6b229a38-f6d5-449e-a529-677e7cafcbc0 is finished
2021-05-16 10:52:25,623 DEBUG awx.main.dispatch task 7e799bf9-7350-41f3-86cd-4895df2395fa is finished
2021-05-16 10:52:25,643 DEBUG awx.main.dispatch task 8e3cb1f1-3773-4026-a55b-48eb43f17f47 is finished
2021-05-16 10:52:31,374 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(912412d2-60b1-4f6d-9717-974d2314f60a, queue=awx_private_queue)
[2021-05-16 10:52:31,374: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(912412d2-60b1-4f6d-9717-974d2314f60a, queue=awx_private_queue)
2021-05-16 10:52:31,390 DEBUG awx.main.dispatch delivered 912412d2-60b1-4f6d-9717-974d2314f60a to worker[198] qsize 0
2021-05-16 10:52:31,395 DEBUG awx.main.dispatch task 912412d2-60b1-4f6d-9717-974d2314f60a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:52:31,396 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:52:41,324 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:52:41,745 DEBUG awx.main.dispatch task 912412d2-60b1-4f6d-9717-974d2314f60a is finished
2021-05-16 10:52:42,087 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9527c098-3de5-41bc-8342-0f858d4dd5ea, queue=awx_private_queue)
[2021-05-16 10:52:42,087: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9527c098-3de5-41bc-8342-0f858d4dd5ea, queue=awx_private_queue)
2021-05-16 10:52:42,109 DEBUG awx.main.dispatch delivered 9527c098-3de5-41bc-8342-0f858d4dd5ea to worker[193] qsize 0
2021-05-16 10:52:42,112 DEBUG awx.main.dispatch task 9527c098-3de5-41bc-8342-0f858d4dd5ea starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:52:42,706 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:52:42,710 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:52:22.985896+00:00
2021-05-16 10:52:42,825 DEBUG awx.main.dispatch task 9527c098-3de5-41bc-8342-0f858d4dd5ea is finished
2021-05-16 10:52:51,375 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(586a9513-83c7-4bcc-b75b-c62c41389d9b, queue=awx_private_queue)
[2021-05-16 10:52:51,375: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(586a9513-83c7-4bcc-b75b-c62c41389d9b, queue=awx_private_queue)
2021-05-16 10:52:51,394 DEBUG awx.main.dispatch delivered 586a9513-83c7-4bcc-b75b-c62c41389d9b to worker[182] qsize 0
2021-05-16 10:52:51,398 DEBUG awx.main.dispatch task 586a9513-83c7-4bcc-b75b-c62c41389d9b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:52:51,400 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:52:55,925 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:52:56,199 DEBUG awx.main.dispatch task 586a9513-83c7-4bcc-b75b-c62c41389d9b is finished
RESULT 2
OKREADY
2021-05-16 10:53:10,767 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5e173e22-3d06-417f-8060-00b3a52d67cc, queue=dev-awx-0)
[2021-05-16 10:53:10,767: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5e173e22-3d06-417f-8060-00b3a52d67cc, queue=dev-awx-0)
2021-05-16 10:53:10,782 WARNING awx.main.dispatch scaling down worker pid:177
2021-05-16 10:53:10,785 WARNING awx.main.dispatch scaling down worker pid:182
2021-05-16 10:53:10,786 WARNING awx.main.dispatch worker exiting gracefully pid:177
2021-05-16 10:53:10,789 WARNING awx.main.dispatch worker exiting gracefully pid:182
2021-05-16 10:53:10,933 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(07dfb1ed-f70c-480e-a9df-baf5d7b64033, queue=dev-awx-0)
[2021-05-16 10:53:10,933: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(07dfb1ed-f70c-480e-a9df-baf5d7b64033, queue=dev-awx-0)
2021-05-16 10:53:11,374 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f5411431-6796-4966-9404-a31a5c27ba88, queue=awx_private_queue)
[2021-05-16 10:53:11,374: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f5411431-6796-4966-9404-a31a5c27ba88, queue=awx_private_queue)
2021-05-16 10:53:12,088 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b5e58852-0fd1-47d4-9699-d9f910b228d3, queue=awx_private_queue)
[2021-05-16 10:53:12,088: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b5e58852-0fd1-47d4-9699-d9f910b228d3, queue=awx_private_queue)
2021-05-16 10:53:21,569 DEBUG awx.main.dispatch delivered 5e173e22-3d06-417f-8060-00b3a52d67cc to worker[184] qsize 0
2021-05-16 10:53:21,572 DEBUG awx.main.dispatch delivered 07dfb1ed-f70c-480e-a9df-baf5d7b64033 to worker[198] qsize 0
2021-05-16 10:53:21,573 DEBUG awx.main.dispatch task 5e173e22-3d06-417f-8060-00b3a52d67cc starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 10:53:21,577 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 10:53:21,575 DEBUG awx.main.dispatch task 07dfb1ed-f70c-480e-a9df-baf5d7b64033 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 10:53:21,582 DEBUG awx.main.dispatch delivered f5411431-6796-4966-9404-a31a5c27ba88 to worker[193] qsize 0
2021-05-16 10:53:21,585 DEBUG awx.main.dispatch task f5411431-6796-4966-9404-a31a5c27ba88 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:53:21,588 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:53:21,593 DEBUG awx.main.dispatch delivered b5e58852-0fd1-47d4-9699-d9f910b228d3 to worker[196] qsize 0
2021-05-16 10:53:21,596 DEBUG awx.main.dispatch task b5e58852-0fd1-47d4-9699-d9f910b228d3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:53:21,781 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:53:22,139 DEBUG awx.main.dispatch task 5e173e22-3d06-417f-8060-00b3a52d67cc is finished
2021-05-16 10:53:22,175 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:53:22,205 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:52:42.706192+00:00
2021-05-16 10:53:22,218 DEBUG awx.main.dispatch task 07dfb1ed-f70c-480e-a9df-baf5d7b64033 is finished
2021-05-16 10:53:22,319 DEBUG awx.main.dispatch task b5e58852-0fd1-47d4-9699-d9f910b228d3 is finished
2021-05-16 10:53:22,625 DEBUG awx.main.dispatch task f5411431-6796-4966-9404-a31a5c27ba88 is finished
2021-05-16 10:53:31,385 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(19b0fb72-5b6d-44da-9555-38a2a9e350dd, queue=awx_private_queue)
[2021-05-16 10:53:31,385: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(19b0fb72-5b6d-44da-9555-38a2a9e350dd, queue=awx_private_queue)
2021-05-16 10:53:31,894 DEBUG awx.main.dispatch delivered 19b0fb72-5b6d-44da-9555-38a2a9e350dd to worker[196] qsize 0
2021-05-16 10:53:32,076 DEBUG awx.main.dispatch task 19b0fb72-5b6d-44da-9555-38a2a9e350dd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:53:32,078 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:53:32,818 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:53:32,987 DEBUG awx.main.dispatch task 19b0fb72-5b6d-44da-9555-38a2a9e350dd is finished
2021-05-16 10:53:42,088 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9483b268-3729-4d86-a797-5fc47dc6d230, queue=awx_private_queue)
[2021-05-16 10:53:42,088: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9483b268-3729-4d86-a797-5fc47dc6d230, queue=awx_private_queue)
2021-05-16 10:53:42,101 DEBUG awx.main.dispatch delivered 9483b268-3729-4d86-a797-5fc47dc6d230 to worker[196] qsize 0
2021-05-16 10:53:42,104 DEBUG awx.main.dispatch task 9483b268-3729-4d86-a797-5fc47dc6d230 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:53:43,453 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:53:43,466 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:53:22.175773+00:00
2021-05-16 10:53:43,520 DEBUG awx.main.dispatch task 9483b268-3729-4d86-a797-5fc47dc6d230 is finished
2021-05-16 10:53:51,385 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(34f35c54-c42b-40b5-93a2-192fbdd5cb5a, queue=awx_private_queue)
[2021-05-16 10:53:51,385: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(34f35c54-c42b-40b5-93a2-192fbdd5cb5a, queue=awx_private_queue)
2021-05-16 10:53:51,405 DEBUG awx.main.dispatch delivered 34f35c54-c42b-40b5-93a2-192fbdd5cb5a to worker[196] qsize 0
2021-05-16 10:53:51,407 DEBUG awx.main.dispatch task 34f35c54-c42b-40b5-93a2-192fbdd5cb5a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:53:51,408 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:53:54,195 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:53:54,231 DEBUG awx.main.dispatch task 34f35c54-c42b-40b5-93a2-192fbdd5cb5a is finished
RESULT 2
OKREADY
2021-05-16 10:54:10,772 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c22e262f-ab1c-4314-b590-871b1e21b441, queue=dev-awx-0)
[2021-05-16 10:54:10,772: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c22e262f-ab1c-4314-b590-871b1e21b441, queue=dev-awx-0)
2021-05-16 10:54:10,933 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(9375e53d-22bd-45f2-93c3-e80cad39faa1, queue=dev-awx-0)
[2021-05-16 10:54:10,933: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(9375e53d-22bd-45f2-93c3-e80cad39faa1, queue=dev-awx-0)
2021-05-16 10:54:11,358 DEBUG awx.main.dispatch delivered c22e262f-ab1c-4314-b590-871b1e21b441 to worker[198] qsize 0
2021-05-16 10:54:11,359 DEBUG awx.main.dispatch task c22e262f-ab1c-4314-b590-871b1e21b441 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 10:54:11,361 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 10:54:11,365 DEBUG awx.main.dispatch delivered 9375e53d-22bd-45f2-93c3-e80cad39faa1 to worker[193] qsize 0
2021-05-16 10:54:11,366 DEBUG awx.main.dispatch task 9375e53d-22bd-45f2-93c3-e80cad39faa1 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 10:54:11,384 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4fab534e-e768-4010-8547-2b40726eedbe, queue=awx_private_queue)
[2021-05-16 10:54:11,384: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4fab534e-e768-4010-8547-2b40726eedbe, queue=awx_private_queue)
2021-05-16 10:54:11,397 DEBUG awx.main.dispatch delivered 4fab534e-e768-4010-8547-2b40726eedbe to worker[184] qsize 0
2021-05-16 10:54:11,399 DEBUG awx.main.dispatch task 4fab534e-e768-4010-8547-2b40726eedbe starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:54:11,402 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:54:11,450 DEBUG awx.main.dispatch task 9375e53d-22bd-45f2-93c3-e80cad39faa1 is finished
2021-05-16 10:54:11,476 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:54:11,537 DEBUG awx.main.dispatch task 4fab534e-e768-4010-8547-2b40726eedbe is finished
2021-05-16 10:54:11,706 DEBUG awx.main.dispatch task c22e262f-ab1c-4314-b590-871b1e21b441 is finished
2021-05-16 10:54:12,087 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8d6b8cd2-d5e8-4b5f-8cbe-18f7d772ef26, queue=awx_private_queue)
[2021-05-16 10:54:12,087: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8d6b8cd2-d5e8-4b5f-8cbe-18f7d772ef26, queue=awx_private_queue)
2021-05-16 10:54:12,101 DEBUG awx.main.dispatch delivered 8d6b8cd2-d5e8-4b5f-8cbe-18f7d772ef26 to worker[184] qsize 0
2021-05-16 10:54:12,103 DEBUG awx.main.dispatch task 8d6b8cd2-d5e8-4b5f-8cbe-18f7d772ef26 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:54:12,148 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:54:12,168 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:53:43.453921+00:00
2021-05-16 10:54:13,019 DEBUG awx.main.dispatch task 8d6b8cd2-d5e8-4b5f-8cbe-18f7d772ef26 is finished
2021-05-16 10:54:31,396 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(44428f9b-69cd-4eef-be37-89dcedc5935a, queue=awx_private_queue)
[2021-05-16 10:54:31,396: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(44428f9b-69cd-4eef-be37-89dcedc5935a, queue=awx_private_queue)
2021-05-16 10:54:38,247 DEBUG awx.main.dispatch delivered 44428f9b-69cd-4eef-be37-89dcedc5935a to worker[198] qsize 0
2021-05-16 10:54:38,530 DEBUG awx.main.dispatch task 44428f9b-69cd-4eef-be37-89dcedc5935a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:54:38,535 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:54:40,107 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:54:41,816 DEBUG awx.main.dispatch task 44428f9b-69cd-4eef-be37-89dcedc5935a is finished
2021-05-16 10:54:42,090 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(cfa05d5b-1a79-430a-a0b8-6963cd350a45, queue=awx_private_queue)
[2021-05-16 10:54:42,090: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(cfa05d5b-1a79-430a-a0b8-6963cd350a45, queue=awx_private_queue)
2021-05-16 10:54:42,108 DEBUG awx.main.dispatch delivered cfa05d5b-1a79-430a-a0b8-6963cd350a45 to worker[198] qsize 0
2021-05-16 10:54:42,111 DEBUG awx.main.dispatch task cfa05d5b-1a79-430a-a0b8-6963cd350a45 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:54:48,338 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:54:49,081 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:54:12.148872+00:00
2021-05-16 10:54:50,144 DEBUG awx.main.dispatch task cfa05d5b-1a79-430a-a0b8-6963cd350a45 is finished
2021-05-16 10:54:51,396 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5247b060-d92c-47f9-905b-045400054223, queue=awx_private_queue)
[2021-05-16 10:54:51,396: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5247b060-d92c-47f9-905b-045400054223, queue=awx_private_queue)
2021-05-16 10:54:51,415 DEBUG awx.main.dispatch delivered 5247b060-d92c-47f9-905b-045400054223 to worker[198] qsize 0
2021-05-16 10:54:51,417 DEBUG awx.main.dispatch task 5247b060-d92c-47f9-905b-045400054223 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:54:51,419 DEBUG awx.main.scheduler Running Tower task manager.
RESULT 2
OKREADY
2021-05-16 10:55:02,199 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:55:03,034 DEBUG awx.main.dispatch task 5247b060-d92c-47f9-905b-045400054223 is finished
2021-05-16 10:55:10,715 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(6db37057-cd93-4a76-a099-f8c5d0bf0b32, queue=dev-awx-0)
[2021-05-16 10:55:10,715: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(6db37057-cd93-4a76-a099-f8c5d0bf0b32, queue=dev-awx-0)
2021-05-16 10:55:10,728 DEBUG awx.main.dispatch delivered 6db37057-cd93-4a76-a099-f8c5d0bf0b32 to worker[196] qsize 0
2021-05-16 10:55:10,732 DEBUG awx.main.dispatch task 6db37057-cd93-4a76-a099-f8c5d0bf0b32 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-05-16 10:55:10,736 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-05-16 10:55:10,772 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(84876610-1304-40e3-815b-13fa34ffe587, queue=dev-awx-0)
[2021-05-16 10:55:10,772: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(84876610-1304-40e3-815b-13fa34ffe587, queue=dev-awx-0)
2021-05-16 10:55:10,934 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(e7341930-9397-4ed0-80ea-dfe731d1d5b7, queue=dev-awx-0)
[2021-05-16 10:55:10,934: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(e7341930-9397-4ed0-80ea-dfe731d1d5b7, queue=dev-awx-0)
2021-05-16 10:55:11,397 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d3ae2e6d-b732-4e6d-b331-9166c0767f01, queue=awx_private_queue)
[2021-05-16 10:55:11,397: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d3ae2e6d-b732-4e6d-b331-9166c0767f01, queue=awx_private_queue)
2021-05-16 10:55:12,088 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(50cefa72-a3ba-4144-9e44-dfb714665f0c, queue=awx_private_queue)
[2021-05-16 10:55:12,088: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(50cefa72-a3ba-4144-9e44-dfb714665f0c, queue=awx_private_queue)
2021-05-16 10:55:12,815 DEBUG awx.main.dispatch delivered 84876610-1304-40e3-815b-13fa34ffe587 to worker[198] qsize 0
2021-05-16 10:55:12,817 DEBUG awx.main.dispatch delivered e7341930-9397-4ed0-80ea-dfe731d1d5b7 to worker[184] qsize 0
2021-05-16 10:55:12,819 DEBUG awx.main.dispatch delivered d3ae2e6d-b732-4e6d-b331-9166c0767f01 to worker[193] qsize 0
2021-05-16 10:55:12,821 DEBUG awx.main.dispatch task e7341930-9397-4ed0-80ea-dfe731d1d5b7 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 10:55:12,823 DEBUG awx.main.dispatch task d3ae2e6d-b732-4e6d-b331-9166c0767f01 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:55:12,829 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:55:12,818 DEBUG awx.main.dispatch task 84876610-1304-40e3-815b-13fa34ffe587 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 10:55:12,834 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 10:55:12,840 WARNING awx.main.dispatch scaling up worker pid:202
2021-05-16 10:55:12,845 DEBUG awx.main.dispatch delivered 50cefa72-a3ba-4144-9e44-dfb714665f0c to worker[202] qsize 0
2021-05-16 10:55:12,850 DEBUG awx.main.dispatch task 50cefa72-a3ba-4144-9e44-dfb714665f0c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:55:13,205 DEBUG awx.main.dispatch task e7341930-9397-4ed0-80ea-dfe731d1d5b7 is finished
2021-05-16 10:55:13,215 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:55:13,221 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:54:48.339208+00:00
2021-05-16 10:55:13,483 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:55:13,516 DEBUG awx.main.dispatch task 50cefa72-a3ba-4144-9e44-dfb714665f0c is finished
2021-05-16 10:55:13,718 DEBUG awx.main.dispatch task d3ae2e6d-b732-4e6d-b331-9166c0767f01 is finished
2021-05-16 10:55:13,761 DEBUG awx.main.dispatch task 6db37057-cd93-4a76-a099-f8c5d0bf0b32 is finished
2021-05-16 10:55:14,089 DEBUG awx.main.dispatch task 84876610-1304-40e3-815b-13fa34ffe587 is finished
2021-05-16 10:55:31,406 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6fadacc6-05fb-436d-a0e4-581678d613a6, queue=awx_private_queue)
[2021-05-16 10:55:31,406: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6fadacc6-05fb-436d-a0e4-581678d613a6, queue=awx_private_queue)
2021-05-16 10:55:31,422 DEBUG awx.main.dispatch delivered 6fadacc6-05fb-436d-a0e4-581678d613a6 to worker[202] qsize 0
2021-05-16 10:55:31,428 DEBUG awx.main.dispatch task 6fadacc6-05fb-436d-a0e4-581678d613a6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:55:31,430 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:55:34,704 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:55:35,352 DEBUG awx.main.dispatch task 6fadacc6-05fb-436d-a0e4-581678d613a6 is finished
2021-05-16 10:55:42,088 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(eaf05f54-7939-4d94-9df8-29c4ba2a8834, queue=awx_private_queue)
[2021-05-16 10:55:42,088: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(eaf05f54-7939-4d94-9df8-29c4ba2a8834, queue=awx_private_queue)
2021-05-16 10:55:43,550 DEBUG awx.main.dispatch delivered eaf05f54-7939-4d94-9df8-29c4ba2a8834 to worker[202] qsize 0
2021-05-16 10:55:45,443 DEBUG awx.main.dispatch task eaf05f54-7939-4d94-9df8-29c4ba2a8834 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:55:48,164 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:55:48,190 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:55:13.215461+00:00
2021-05-16 10:55:48,380 DEBUG awx.main.dispatch task eaf05f54-7939-4d94-9df8-29c4ba2a8834 is finished
2021-05-16 10:55:51,411 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1207854b-e1be-4c92-87c8-f82f50cc5382, queue=awx_private_queue)
[2021-05-16 10:55:51,411: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1207854b-e1be-4c92-87c8-f82f50cc5382, queue=awx_private_queue)
2021-05-16 10:55:51,445 DEBUG awx.main.dispatch delivered 1207854b-e1be-4c92-87c8-f82f50cc5382 to worker[198] qsize 0
2021-05-16 10:55:51,451 DEBUG awx.main.dispatch task 1207854b-e1be-4c92-87c8-f82f50cc5382 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:55:51,456 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:56:01,245 DEBUG awx.main.scheduler Starting Scheduler
RESULT 2
OKREADY
2021-05-16 10:56:02,225 DEBUG awx.main.dispatch task 1207854b-e1be-4c92-87c8-f82f50cc5382 is finished
2021-05-16 10:56:10,777 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(76a44cca-7b9b-4b1e-98fa-8a7d4d56c5da, queue=dev-awx-0)
[2021-05-16 10:56:10,777: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(76a44cca-7b9b-4b1e-98fa-8a7d4d56c5da, queue=dev-awx-0)
2021-05-16 10:56:10,799 WARNING awx.main.dispatch scaling down worker pid:184
2021-05-16 10:56:10,804 WARNING awx.main.dispatch worker exiting gracefully pid:184
2021-05-16 10:56:10,934 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(1b551f92-879f-4bc9-a19e-9698d4681d5d, queue=dev-awx-0)
[2021-05-16 10:56:10,934: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(1b551f92-879f-4bc9-a19e-9698d4681d5d, queue=dev-awx-0)
2021-05-16 10:56:11,407 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7ffe4bed-d1a0-4e87-8510-927429592289, queue=awx_private_queue)
[2021-05-16 10:56:11,407: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7ffe4bed-d1a0-4e87-8510-927429592289, queue=awx_private_queue)
2021-05-16 10:56:12,088 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(71d9dbd8-cc40-47e2-8159-07e316168284, queue=awx_private_queue)
[2021-05-16 10:56:12,088: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(71d9dbd8-cc40-47e2-8159-07e316168284, queue=awx_private_queue)
2021-05-16 10:56:31,417 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(184b05ce-ddcd-464c-ad3a-144b1cf0d5e9, queue=awx_private_queue)
[2021-05-16 10:56:31,417: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(184b05ce-ddcd-464c-ad3a-144b1cf0d5e9, queue=awx_private_queue)
2021-05-16 10:56:36,386 DEBUG awx.main.dispatch delivered 76a44cca-7b9b-4b1e-98fa-8a7d4d56c5da to worker[202] qsize 0
2021-05-16 10:56:36,387 DEBUG awx.main.dispatch delivered 1b551f92-879f-4bc9-a19e-9698d4681d5d to worker[196] qsize 0
2021-05-16 10:56:36,388 DEBUG awx.main.dispatch task 76a44cca-7b9b-4b1e-98fa-8a7d4d56c5da starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 10:56:36,390 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 10:56:36,390 DEBUG awx.main.dispatch delivered 7ffe4bed-d1a0-4e87-8510-927429592289 to worker[193] qsize 0
2021-05-16 10:56:36,392 DEBUG awx.main.dispatch delivered 71d9dbd8-cc40-47e2-8159-07e316168284 to worker[198] qsize 0
2021-05-16 10:56:36,390 DEBUG awx.main.dispatch task 1b551f92-879f-4bc9-a19e-9698d4681d5d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 10:56:36,399 DEBUG awx.main.dispatch task 71d9dbd8-cc40-47e2-8159-07e316168284 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:56:36,394 DEBUG awx.main.dispatch task 7ffe4bed-d1a0-4e87-8510-927429592289 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:56:36,401 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:56:36,400 WARNING awx.main.dispatch scaling up worker pid:205
2021-05-16 10:56:36,405 DEBUG awx.main.dispatch delivered 184b05ce-ddcd-464c-ad3a-144b1cf0d5e9 to worker[205] qsize 0
2021-05-16 10:56:36,412 DEBUG awx.main.dispatch task 184b05ce-ddcd-464c-ad3a-144b1cf0d5e9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:56:36,415 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:56:37,864 DEBUG awx.main.dispatch task 1b551f92-879f-4bc9-a19e-9698d4681d5d is finished
2021-05-16 10:56:37,986 DEBUG awx.main.dispatch task 76a44cca-7b9b-4b1e-98fa-8a7d4d56c5da is finished
2021-05-16 10:56:38,058 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:56:38,061 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:56:38,064 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-16 10:56:38,065 DEBUG awx.main.dispatch task 184b05ce-ddcd-464c-ad3a-144b1cf0d5e9 is finished
2021-05-16 10:56:38,087 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:55:48.164973+00:00
2021-05-16 10:56:38,473 DEBUG awx.main.dispatch task 71d9dbd8-cc40-47e2-8159-07e316168284 is finished
2021-05-16 10:56:38,620 DEBUG awx.main.dispatch task 7ffe4bed-d1a0-4e87-8510-927429592289 is finished
2021-05-16 10:56:42,088 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(285d66ea-3d79-4a9d-867c-e2963615d06f, queue=awx_private_queue)
[2021-05-16 10:56:42,088: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(285d66ea-3d79-4a9d-867c-e2963615d06f, queue=awx_private_queue)
2021-05-16 10:56:42,101 DEBUG awx.main.dispatch delivered 285d66ea-3d79-4a9d-867c-e2963615d06f to worker[205] qsize 0
2021-05-16 10:56:42,103 DEBUG awx.main.dispatch task 285d66ea-3d79-4a9d-867c-e2963615d06f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:56:43,896 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:56:43,935 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:56:38.058600+00:00
2021-05-16 10:56:44,445 DEBUG awx.main.dispatch task 285d66ea-3d79-4a9d-867c-e2963615d06f is finished
RESULT 2
OKREADY
2021-05-16 10:56:51,416 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d8d3bfe9-37a3-4e13-97e8-42975a7ede69, queue=awx_private_queue)
[2021-05-16 10:56:51,416: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d8d3bfe9-37a3-4e13-97e8-42975a7ede69, queue=awx_private_queue)
2021-05-16 10:57:11,565 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c433f7a7-595e-46bb-b9be-2d5f68177e41, queue=dev-awx-0)
[2021-05-16 10:57:11,565: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c433f7a7-595e-46bb-b9be-2d5f68177e41, queue=dev-awx-0)
2021-05-16 10:57:17,141 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(92172f90-c457-4ea8-b7e9-04ef03f9a621, queue=dev-awx-0)
[2021-05-16 10:57:17,141: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(92172f90-c457-4ea8-b7e9-04ef03f9a621, queue=dev-awx-0)
2021-05-16 10:57:17,157 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b26a2b1a-b8b3-4443-a9c1-eccf2bc82bbc, queue=awx_private_queue)
[2021-05-16 10:57:17,157: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b26a2b1a-b8b3-4443-a9c1-eccf2bc82bbc, queue=awx_private_queue)
2021-05-16 10:57:17,178 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e91bee62-9b85-4590-940c-8bcfe4952712, queue=awx_private_queue)
[2021-05-16 10:57:17,178: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e91bee62-9b85-4590-940c-8bcfe4952712, queue=awx_private_queue)
2021-05-16 10:57:11,563 DEBUG awx.main.dispatch delivered d8d3bfe9-37a3-4e13-97e8-42975a7ede69 to worker[205] qsize 0
2021-05-16 10:57:17,232 WARNING awx.main.dispatch scaling down worker pid:193
2021-05-16 10:57:17,233 DEBUG awx.main.dispatch task d8d3bfe9-37a3-4e13-97e8-42975a7ede69 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:57:17,237 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:57:17,235 WARNING awx.main.dispatch worker exiting gracefully pid:193
2021-05-16 10:57:17,420 DEBUG awx.main.dispatch delivered c433f7a7-595e-46bb-b9be-2d5f68177e41 to worker[202] qsize 0
2021-05-16 10:57:17,422 DEBUG awx.main.dispatch delivered 92172f90-c457-4ea8-b7e9-04ef03f9a621 to worker[198] qsize 0
2021-05-16 10:57:17,424 DEBUG awx.main.dispatch delivered b26a2b1a-b8b3-4443-a9c1-eccf2bc82bbc to worker[196] qsize 0
2021-05-16 10:57:17,424 DEBUG awx.main.dispatch task c433f7a7-595e-46bb-b9be-2d5f68177e41 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 10:57:17,430 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 10:57:17,426 DEBUG awx.main.dispatch task 92172f90-c457-4ea8-b7e9-04ef03f9a621 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 10:57:17,429 DEBUG awx.main.dispatch task b26a2b1a-b8b3-4443-a9c1-eccf2bc82bbc starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:57:17,437 WARNING awx.main.dispatch scaling up worker pid:208
2021-05-16 10:57:17,445 DEBUG awx.main.dispatch delivered e91bee62-9b85-4590-940c-8bcfe4952712 to worker[208] qsize 0
2021-05-16 10:57:17,452 DEBUG awx.main.dispatch task e91bee62-9b85-4590-940c-8bcfe4952712 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:57:17,455 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:57:17,561 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:57:18,194 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:57:18,257 DEBUG awx.main.dispatch task 92172f90-c457-4ea8-b7e9-04ef03f9a621 is finished
2021-05-16 10:57:18,277 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:56:43.896360+00:00
2021-05-16 10:57:18,599 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-16 10:57:18,600 DEBUG awx.main.dispatch task e91bee62-9b85-4590-940c-8bcfe4952712 is finished
2021-05-16 10:57:21,661 DEBUG awx.main.dispatch task b26a2b1a-b8b3-4443-a9c1-eccf2bc82bbc is finished
2021-05-16 10:57:22,053 DEBUG awx.main.dispatch task c433f7a7-595e-46bb-b9be-2d5f68177e41 is finished
2021-05-16 10:57:22,084 DEBUG awx.main.dispatch task d8d3bfe9-37a3-4e13-97e8-42975a7ede69 is finished
2021-05-16 10:57:37,178 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c02adbef-2807-4abb-b1e3-e7d047484fea, queue=awx_private_queue)
[2021-05-16 10:57:37,178: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c02adbef-2807-4abb-b1e3-e7d047484fea, queue=awx_private_queue)
2021-05-16 10:57:37,196 DEBUG awx.main.dispatch delivered c02adbef-2807-4abb-b1e3-e7d047484fea to worker[196] qsize 0
2021-05-16 10:57:37,198 DEBUG awx.main.dispatch task c02adbef-2807-4abb-b1e3-e7d047484fea starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:57:37,200 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:57:37,314 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:57:37,887 DEBUG awx.main.dispatch task c02adbef-2807-4abb-b1e3-e7d047484fea is finished
2021-05-16 10:57:47,157 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(85e32ef3-8fb6-4753-90d8-942d29748dde, queue=awx_private_queue)
[2021-05-16 10:57:47,157: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(85e32ef3-8fb6-4753-90d8-942d29748dde, queue=awx_private_queue)
2021-05-16 10:57:47,172 DEBUG awx.main.dispatch delivered 85e32ef3-8fb6-4753-90d8-942d29748dde to worker[208] qsize 0
2021-05-16 10:57:47,174 DEBUG awx.main.dispatch task 85e32ef3-8fb6-4753-90d8-942d29748dde starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:57:47,293 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:57:47,558 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:57:18.194879+00:00
2021-05-16 10:57:47,783 DEBUG awx.main.dispatch task 85e32ef3-8fb6-4753-90d8-942d29748dde is finished
2021-05-16 10:57:57,180 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2482e4ea-6831-4a74-a282-e88afc0ae395, queue=awx_private_queue)
[2021-05-16 10:57:57,180: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2482e4ea-6831-4a74-a282-e88afc0ae395, queue=awx_private_queue)
2021-05-16 10:57:57,201 DEBUG awx.main.dispatch delivered 2482e4ea-6831-4a74-a282-e88afc0ae395 to worker[208] qsize 0
2021-05-16 10:57:57,206 DEBUG awx.main.dispatch task 2482e4ea-6831-4a74-a282-e88afc0ae395 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:57:57,208 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:57:57,260 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:57:57,393 DEBUG awx.main.dispatch task 2482e4ea-6831-4a74-a282-e88afc0ae395 is finished
RESULT 2
OKREADY
2021-05-16 10:58:11,568 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e2b56ce8-7502-4728-80c6-b56eeeeef00f, queue=dev-awx-0)
[2021-05-16 10:58:11,568: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e2b56ce8-7502-4728-80c6-b56eeeeef00f, queue=dev-awx-0)
2021-05-16 10:58:11,582 WARNING awx.main.dispatch scaling down worker pid:196
2021-05-16 10:58:11,585 WARNING awx.main.dispatch worker exiting gracefully pid:196
2021-05-16 10:58:11,688 DEBUG awx.main.dispatch delivered e2b56ce8-7502-4728-80c6-b56eeeeef00f to worker[205] qsize 0
2021-05-16 10:58:11,691 DEBUG awx.main.dispatch task e2b56ce8-7502-4728-80c6-b56eeeeef00f starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 10:58:11,695 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 10:58:11,726 DEBUG awx.main.dispatch task e2b56ce8-7502-4728-80c6-b56eeeeef00f is finished
2021-05-16 10:58:17,145 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5c1c08fd-9034-474e-95f5-63d1e26c22fa, queue=dev-awx-0)
[2021-05-16 10:58:17,145: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5c1c08fd-9034-474e-95f5-63d1e26c22fa, queue=dev-awx-0)
2021-05-16 10:58:17,348 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ee613cd4-0453-41a8-9d9f-a9a5375b7495, queue=awx_private_queue)
[2021-05-16 10:58:17,348: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ee613cd4-0453-41a8-9d9f-a9a5375b7495, queue=awx_private_queue)
2021-05-16 10:58:17,347 DEBUG awx.main.dispatch delivered 5c1c08fd-9034-474e-95f5-63d1e26c22fa to worker[198] qsize 0
2021-05-16 10:58:17,368 DEBUG awx.main.dispatch delivered ee613cd4-0453-41a8-9d9f-a9a5375b7495 to worker[205] qsize 0
2021-05-16 10:58:17,371 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(430bdc60-39cd-40e4-b2c2-c4b0b24605f6, queue=awx_private_queue)
[2021-05-16 10:58:17,371: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(430bdc60-39cd-40e4-b2c2-c4b0b24605f6, queue=awx_private_queue)
2021-05-16 10:58:17,370 DEBUG awx.main.dispatch task ee613cd4-0453-41a8-9d9f-a9a5375b7495 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:58:17,384 DEBUG awx.main.dispatch delivered 430bdc60-39cd-40e4-b2c2-c4b0b24605f6 to worker[208] qsize 0
2021-05-16 10:58:17,417 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:58:17,432 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:57:47.293710+00:00
2021-05-16 10:58:17,531 DEBUG awx.main.dispatch task ee613cd4-0453-41a8-9d9f-a9a5375b7495 is finished
2021-05-16 10:58:17,352 DEBUG awx.main.dispatch task 5c1c08fd-9034-474e-95f5-63d1e26c22fa starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 10:58:17,624 DEBUG awx.main.dispatch task 5c1c08fd-9034-474e-95f5-63d1e26c22fa is finished
2021-05-16 10:58:17,387 DEBUG awx.main.dispatch task 430bdc60-39cd-40e4-b2c2-c4b0b24605f6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:58:17,637 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:58:17,656 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:58:18,104 DEBUG awx.main.dispatch task 430bdc60-39cd-40e4-b2c2-c4b0b24605f6 is finished
2021-05-16 10:58:37,382 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5de41737-f1a7-484b-ab54-0f576c0bb37c, queue=awx_private_queue)
[2021-05-16 10:58:37,382: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5de41737-f1a7-484b-ab54-0f576c0bb37c, queue=awx_private_queue)
2021-05-16 10:58:37,399 DEBUG awx.main.dispatch delivered 5de41737-f1a7-484b-ab54-0f576c0bb37c to worker[202] qsize 0
2021-05-16 10:58:37,404 DEBUG awx.main.dispatch task 5de41737-f1a7-484b-ab54-0f576c0bb37c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:58:37,410 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 10:58:39,924 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 10:58:40,090 DEBUG awx.main.dispatch task 5de41737-f1a7-484b-ab54-0f576c0bb37c is finished
2021-05-16 10:58:47,348 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dea109d7-a212-4c1c-97fa-76533318ea96, queue=awx_private_queue)
[2021-05-16 10:58:47,348: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dea109d7-a212-4c1c-97fa-76533318ea96, queue=awx_private_queue)
2021-05-16 10:58:47,362 DEBUG awx.main.dispatch delivered dea109d7-a212-4c1c-97fa-76533318ea96 to worker[202] qsize 0
2021-05-16 10:58:47,365 DEBUG awx.main.dispatch task dea109d7-a212-4c1c-97fa-76533318ea96 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:58:47,658 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 10:58:47,687 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:58:17.417882+00:00
2021-05-16 10:58:47,757 DEBUG awx.main.dispatch task dea109d7-a212-4c1c-97fa-76533318ea96 is finished
2021-05-16 10:58:57,380 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3c627a54-184a-4927-983b-a823b5723d14, queue=awx_private_queue)
[2021-05-16 10:58:57,380: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3c627a54-184a-4927-983b-a823b5723d14, queue=awx_private_queue)
2021-05-16 10:58:57,395 DEBUG awx.main.dispatch delivered 3c627a54-184a-4927-983b-a823b5723d14 to worker[208] qsize 0
2021-05-16 10:58:57,399 DEBUG awx.main.dispatch task 3c627a54-184a-4927-983b-a823b5723d14 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:58:57,401 DEBUG awx.main.scheduler Running Tower task manager.
RESULT 2
OKREADY
2021-05-16 10:59:11,576 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(0a577cd6-e0cb-4c7f-82fd-460b0e74179b, queue=dev-awx-0)
[2021-05-16 10:59:11,576: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(0a577cd6-e0cb-4c7f-82fd-460b0e74179b, queue=dev-awx-0)
2021-05-16 10:59:12,460 DEBUG awx.main.dispatch delivered 0a577cd6-e0cb-4c7f-82fd-460b0e74179b to worker[198] qsize 0
2021-05-16 10:59:12,466 DEBUG awx.main.dispatch task 0a577cd6-e0cb-4c7f-82fd-460b0e74179b starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 10:59:12,469 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 10:59:17,143 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(fae3cfd4-b3fa-4ad8-9909-c42a3d79b38e, queue=dev-awx-0)
[2021-05-16 10:59:17,143: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(fae3cfd4-b3fa-4ad8-9909-c42a3d79b38e, queue=dev-awx-0)
2021-05-16 10:59:17,165 DEBUG awx.main.dispatch delivered fae3cfd4-b3fa-4ad8-9909-c42a3d79b38e to worker[205] qsize 0
2021-05-16 10:59:17,169 DEBUG awx.main.dispatch task fae3cfd4-b3fa-4ad8-9909-c42a3d79b38e starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 10:59:17,348 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ff9e7ffd-fadd-4b51-9169-6633d16dd865, queue=awx_private_queue)
[2021-05-16 10:59:17,348: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ff9e7ffd-fadd-4b51-9169-6633d16dd865, queue=awx_private_queue)
2021-05-16 10:59:17,367 DEBUG awx.main.dispatch delivered ff9e7ffd-fadd-4b51-9169-6633d16dd865 to worker[202] qsize 0
2021-05-16 10:59:17,369 DEBUG awx.main.dispatch task ff9e7ffd-fadd-4b51-9169-6633d16dd865 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 10:59:17,380 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f426610a-2ce4-4086-a532-b7273aab5b96, queue=awx_private_queue)
[2021-05-16 10:59:17,380: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f426610a-2ce4-4086-a532-b7273aab5b96, queue=awx_private_queue)
2021-05-16 10:59:17,418 WARNING awx.main.dispatch scaling up worker pid:211
2021-05-16 10:59:17,426 DEBUG awx.main.dispatch delivered f426610a-2ce4-4086-a532-b7273aab5b96 to worker[211] qsize 0
2021-05-16 10:59:17,433 DEBUG awx.main.dispatch task f426610a-2ce4-4086-a532-b7273aab5b96 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 10:59:17,437 DEBUG awx.main.scheduler Running Tower task manager.
RESULT 2
OKREADY
2021-05-16 11:00:24,843 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 11:00:25,409 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-16 10:59:37,391 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(abc2c626-e2e5-416e-a57e-f2c68b750787, queue=awx_private_queue)
[2021-05-16 10:59:37,391: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(abc2c626-e2e5-416e-a57e-f2c68b750787, queue=awx_private_queue)
2021-05-16 11:00:25,410 DEBUG awx.main.dispatch task 3c627a54-184a-4927-983b-a823b5723d14 is finished
2021-05-16 11:00:36,866 DEBUG awx.main.tasks Starting periodic scheduler
RESULT 2
OKREADY
2021-05-16 11:01:38,992 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 10:58:47.659200+00:00
RESULT 2
OKREADY
2021-05-16 11:00:25,447 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ebfdae47-6f12-4f87-a769-41166a10794b, queue=awx_private_queue)
[2021-05-16 11:00:25,447: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ebfdae47-6f12-4f87-a769-41166a10794b, queue=awx_private_queue)
2021-05-16 11:02:25,134 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(7dc63358-cd87-4567-8adf-d4f74fd505a2, queue=dev-awx-0)
[2021-05-16 11:02:25,134: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(7dc63358-cd87-4567-8adf-d4f74fd505a2, queue=dev-awx-0)
2021-05-16 11:02:25,162 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(935ad7fc-cb25-4ae0-a9e0-7dfbac8210e5, queue=dev-awx-0)
[2021-05-16 11:02:25,162: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(935ad7fc-cb25-4ae0-a9e0-7dfbac8210e5, queue=dev-awx-0)
2021-05-16 11:02:25,174 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0b8cb281-40b9-425b-ac3f-b4d4f9bf6831, queue=awx_private_queue)
[2021-05-16 11:02:25,174: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0b8cb281-40b9-425b-ac3f-b4d4f9bf6831, queue=awx_private_queue)
2021-05-16 11:02:26,962 DEBUG awx.main.dispatch task ff9e7ffd-fadd-4b51-9169-6633d16dd865 is finished
2021-05-16 11:01:39,050 DEBUG awx.main.dispatch task fae3cfd4-b3fa-4ad8-9909-c42a3d79b38e is finished
2021-05-16 11:02:45,188 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(90aa2171-0ab5-4a76-8f3e-1999f796a3da, queue=awx_private_queue)
[2021-05-16 11:02:45,188: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(90aa2171-0ab5-4a76-8f3e-1999f796a3da, queue=awx_private_queue)
2021-05-16 11:02:45,201 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5f6e1cac-1177-42b8-a4db-270df10dae11, queue=awx_private_queue)
[2021-05-16 11:02:45,201: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5f6e1cac-1177-42b8-a4db-270df10dae11, queue=awx_private_queue)
RESULT 2
OKREADY
2021-05-16 11:03:05,192 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(90f8f927-2871-4616-be41-b0dd5c0a9c96, queue=awx_private_queue)
[2021-05-16 11:03:05,192: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(90f8f927-2871-4616-be41-b0dd5c0a9c96, queue=awx_private_queue)
2021-05-16 11:03:15,202 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(c6f85672-94ec-4f22-8829-8d3f2ee3a976, queue=awx_private_queue)
[2021-05-16 11:03:15,202: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(c6f85672-94ec-4f22-8829-8d3f2ee3a976, queue=awx_private_queue)
2021-05-16 11:03:25,140 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(16f66d69-6c5b-4475-99df-f22677715505, queue=dev-awx-0)
[2021-05-16 11:03:25,140: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(16f66d69-6c5b-4475-99df-f22677715505, queue=dev-awx-0)
2021-05-16 11:03:25,167 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8076b9ea-0510-41c9-ba40-0f24a9bf8c05, queue=dev-awx-0)
[2021-05-16 11:03:25,167: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8076b9ea-0510-41c9-ba40-0f24a9bf8c05, queue=dev-awx-0)
2021-05-16 11:03:25,191 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fa8e912a-4c7b-4199-8584-edc9f39857b2, queue=awx_private_queue)
[2021-05-16 11:03:25,191: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fa8e912a-4c7b-4199-8584-edc9f39857b2, queue=awx_private_queue)
RESULT 2
OKREADY
2021-05-16 11:00:25,461 WARNING awx.main.dispatch scaling up worker pid:213
2021-05-16 11:04:12,722 DEBUG awx.main.dispatch delivered abc2c626-e2e5-416e-a57e-f2c68b750787 to worker[202] qsize 0
2021-05-16 11:04:12,730 DEBUG awx.main.dispatch delivered ebfdae47-6f12-4f87-a769-41166a10794b to worker[213] qsize 0
2021-05-16 11:04:12,730 DEBUG awx.main.dispatch task abc2c626-e2e5-416e-a57e-f2c68b750787 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:04:12,735 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:04:12,734 WARNING awx.main.dispatch scaling down worker pid:205
2021-05-16 11:04:12,737 WARNING awx.main.dispatch scaling down worker pid:208
2021-05-16 11:04:12,739 WARNING awx.main.dispatch worker exiting gracefully pid:208
2021-05-16 11:04:12,739 WARNING awx.main.dispatch worker exiting gracefully pid:205
2021-05-16 11:04:12,736 DEBUG awx.main.dispatch task ebfdae47-6f12-4f87-a769-41166a10794b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:04:12,812 DEBUG awx.main.dispatch task 0a577cd6-e0cb-4c7f-82fd-460b0e74179b is finished
2021-05-16 11:04:12,862 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:04:12,868 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-16 11:04:12,868 DEBUG awx.main.dispatch task abc2c626-e2e5-416e-a57e-f2c68b750787 is finished
2021-05-16 11:04:12,868 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 11:00:36.868940+00:00
2021-05-16 11:04:12,902 DEBUG awx.main.dispatch task ebfdae47-6f12-4f87-a769-41166a10794b is finished
2021-05-16 11:04:12,954 DEBUG awx.main.dispatch delivered 7dc63358-cd87-4567-8adf-d4f74fd505a2 to worker[213] qsize 0
2021-05-16 11:04:12,958 DEBUG awx.main.dispatch delivered 935ad7fc-cb25-4ae0-a9e0-7dfbac8210e5 to worker[198] qsize 0
2021-05-16 11:04:12,959 DEBUG awx.main.dispatch task 7dc63358-cd87-4567-8adf-d4f74fd505a2 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 11:04:12,960 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 11:04:12,962 DEBUG awx.main.dispatch task 935ad7fc-cb25-4ae0-a9e0-7dfbac8210e5 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:04:12,963 DEBUG awx.main.dispatch delivered 0b8cb281-40b9-425b-ac3f-b4d4f9bf6831 to worker[202] qsize 0
2021-05-16 11:04:12,965 DEBUG awx.main.dispatch task 0b8cb281-40b9-425b-ac3f-b4d4f9bf6831 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:04:12,966 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:04:12,977 WARNING awx.main.dispatch scaling up worker pid:216
2021-05-16 11:04:12,984 DEBUG awx.main.dispatch delivered 90aa2171-0ab5-4a76-8f3e-1999f796a3da to worker[216] qsize 0
2021-05-16 11:04:12,991 DEBUG awx.main.dispatch task 90aa2171-0ab5-4a76-8f3e-1999f796a3da starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:04:12,994 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:04:12,994 WARNING awx.main.dispatch scaling up worker pid:218
2021-05-16 11:04:13,002 DEBUG awx.main.dispatch delivered 5f6e1cac-1177-42b8-a4db-270df10dae11 to worker[218] qsize 0
2021-05-16 11:04:13,008 DEBUG awx.main.dispatch task 5f6e1cac-1177-42b8-a4db-270df10dae11 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:04:13,017 WARNING awx.main.dispatch scaling up worker pid:220
2021-05-16 11:04:13,023 DEBUG awx.main.dispatch delivered 90f8f927-2871-4616-be41-b0dd5c0a9c96 to worker[220] qsize 0
2021-05-16 11:04:13,028 DEBUG awx.main.dispatch task 90f8f927-2871-4616-be41-b0dd5c0a9c96 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:04:13,030 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:04:13,033 WARNING awx.main.dispatch scaling up worker pid:222
2021-05-16 11:04:13,040 DEBUG awx.main.dispatch delivered c6f85672-94ec-4f22-8829-8d3f2ee3a976 to worker[222] qsize 0
2021-05-16 11:04:13,045 DEBUG awx.main.dispatch task c6f85672-94ec-4f22-8829-8d3f2ee3a976 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:04:14,905 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:04:14,915 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 11:04:12.862895+00:00
2021-05-16 11:04:14,933 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-16 11:04:14,934 DEBUG awx.main.dispatch task 90aa2171-0ab5-4a76-8f3e-1999f796a3da is finished
2021-05-16 11:04:14,949 DEBUG awx.main.dispatch task c6f85672-94ec-4f22-8829-8d3f2ee3a976 is finished
2021-05-16 11:04:15,905 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-16 11:04:15,905 DEBUG awx.main.dispatch task 0b8cb281-40b9-425b-ac3f-b4d4f9bf6831 is finished
2021-05-16 11:03:45,194 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(75775fa2-d5a9-447d-b055-3a23fa663b2e, queue=awx_private_queue)
[2021-05-16 11:03:45,194: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(75775fa2-d5a9-447d-b055-3a23fa663b2e, queue=awx_private_queue)
2021-05-16 11:04:19,212 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(5ec51c3a-8960-47b4-b90a-9dec59a9f301, queue=awx_private_queue)
[2021-05-16 11:04:19,212: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(5ec51c3a-8960-47b4-b90a-9dec59a9f301, queue=awx_private_queue)
2021-05-16 11:04:24,887 DEBUG awx.main.dispatch task f426610a-2ce4-4086-a532-b7273aab5b96 is finished
2021-05-16 11:04:25,134 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(ae474578-1e53-4dbc-8603-b7734de0df95, queue=dev-awx-0)
[2021-05-16 11:04:25,134: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(ae474578-1e53-4dbc-8603-b7734de0df95, queue=dev-awx-0)
2021-05-16 11:04:25,167 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d65d385a-dcae-4c7f-90c6-857a9ef4687a, queue=dev-awx-0)
[2021-05-16 11:04:25,167: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d65d385a-dcae-4c7f-90c6-857a9ef4687a, queue=dev-awx-0)
2021-05-16 11:04:25,182 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7f37057c-34a5-4a19-a2de-66000dda068d, queue=awx_private_queue)
[2021-05-16 11:04:25,182: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7f37057c-34a5-4a19-a2de-66000dda068d, queue=awx_private_queue)
2021-05-16 11:04:26,338 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 11:04:27,815 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:04:27,855 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 11:04:14.905665+00:00
2021-05-16 11:04:27,964 DEBUG awx.main.dispatch task 5f6e1cac-1177-42b8-a4db-270df10dae11 is finished
2021-05-16 11:04:28,402 DEBUG awx.main.dispatch task 935ad7fc-cb25-4ae0-a9e0-7dfbac8210e5 is finished
2021-05-16 11:04:28,607 DEBUG awx.main.dispatch task 90f8f927-2871-4616-be41-b0dd5c0a9c96 is finished
2021-05-16 11:04:29,499 DEBUG awx.main.dispatch delivered 16f66d69-6c5b-4475-99df-f22677715505 to worker[211] qsize 0
2021-05-16 11:04:29,501 DEBUG awx.main.dispatch delivered 8076b9ea-0510-41c9-ba40-0f24a9bf8c05 to worker[220] qsize 0
2021-05-16 11:04:29,503 DEBUG awx.main.dispatch delivered fa8e912a-4c7b-4199-8584-edc9f39857b2 to worker[216] qsize 0
2021-05-16 11:04:29,503 DEBUG awx.main.dispatch task 8076b9ea-0510-41c9-ba40-0f24a9bf8c05 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:04:29,502 DEBUG awx.main.dispatch task 16f66d69-6c5b-4475-99df-f22677715505 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 11:04:29,504 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 11:04:29,504 DEBUG awx.main.dispatch delivered 75775fa2-d5a9-447d-b055-3a23fa663b2e to worker[202] qsize 0
2021-05-16 11:04:29,505 DEBUG awx.main.dispatch delivered 5ec51c3a-8960-47b4-b90a-9dec59a9f301 to worker[218] qsize 0
2021-05-16 11:04:29,505 DEBUG awx.main.dispatch task fa8e912a-4c7b-4199-8584-edc9f39857b2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:04:29,506 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:04:29,506 DEBUG awx.main.dispatch task 75775fa2-d5a9-447d-b055-3a23fa663b2e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:04:29,508 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:04:29,509 WARNING awx.main.dispatch scaling down worker pid:198
2021-05-16 11:04:29,510 DEBUG awx.main.dispatch task 5ec51c3a-8960-47b4-b90a-9dec59a9f301 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:04:29,511 WARNING awx.main.dispatch scaling down worker pid:222
2021-05-16 11:04:29,511 WARNING awx.main.dispatch worker exiting gracefully pid:198
2021-05-16 11:04:29,513 WARNING awx.main.dispatch worker exiting gracefully pid:222
2021-05-16 11:04:29,730 WARNING awx.main.dispatch scaling up worker pid:229
2021-05-16 11:04:29,736 DEBUG awx.main.dispatch delivered ae474578-1e53-4dbc-8603-b7734de0df95 to worker[229] qsize 0
2021-05-16 11:04:29,740 DEBUG awx.main.dispatch task ae474578-1e53-4dbc-8603-b7734de0df95 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-16 11:04:29,742 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-16 11:04:29,750 WARNING awx.main.dispatch scaling up worker pid:231
2021-05-16 11:04:29,757 DEBUG awx.main.dispatch delivered d65d385a-dcae-4c7f-90c6-857a9ef4687a to worker[231] qsize 0
2021-05-16 11:04:29,760 DEBUG awx.main.dispatch task d65d385a-dcae-4c7f-90c6-857a9ef4687a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:04:29,766 WARNING awx.main.dispatch scaling up worker pid:233
2021-05-16 11:04:29,770 DEBUG awx.main.dispatch delivered 7f37057c-34a5-4a19-a2de-66000dda068d to worker[233] qsize 0
2021-05-16 11:04:29,775 DEBUG awx.main.dispatch task 7f37057c-34a5-4a19-a2de-66000dda068d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:04:29,778 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:04:29,857 DEBUG awx.main.dispatch task 7dc63358-cd87-4567-8adf-d4f74fd505a2 is finished
2021-05-16 11:04:30,652 DEBUG awx.main.dispatch task d65d385a-dcae-4c7f-90c6-857a9ef4687a is finished
2021-05-16 11:04:30,749 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:04:30,773 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 11:04:27.816184+00:00
2021-05-16 11:04:30,927 DEBUG awx.main.dispatch task ae474578-1e53-4dbc-8603-b7734de0df95 is finished
2021-05-16 11:04:30,930 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 11:04:30,968 DEBUG awx.main.dispatch task 16f66d69-6c5b-4475-99df-f22677715505 is finished
2021-05-16 11:04:31,174 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-16 11:04:31,175 DEBUG awx.main.dispatch task 75775fa2-d5a9-447d-b055-3a23fa663b2e is finished
2021-05-16 11:04:33,428 DEBUG awx.main.dispatch task 5ec51c3a-8960-47b4-b90a-9dec59a9f301 is finished
2021-05-16 11:04:34,462 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
2021-05-16 11:04:34,462 DEBUG awx.main.dispatch task fa8e912a-4c7b-4199-8584-edc9f39857b2 is finished
2021-05-16 11:04:35,060 DEBUG awx.main.dispatch task 8076b9ea-0510-41c9-ba40-0f24a9bf8c05 is finished
2021-05-16 11:04:36,466 DEBUG awx.main.dispatch task 7f37057c-34a5-4a19-a2de-66000dda068d is finished
2021-05-16 11:04:45,191 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(82a6c374-7ce7-4cb8-aa2a-dcd256d24f8b, queue=awx_private_queue)
[2021-05-16 11:04:45,191: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(82a6c374-7ce7-4cb8-aa2a-dcd256d24f8b, queue=awx_private_queue)
2021-05-16 11:04:45,205 DEBUG awx.main.dispatch delivered 82a6c374-7ce7-4cb8-aa2a-dcd256d24f8b to worker[216] qsize 0
2021-05-16 11:04:45,207 DEBUG awx.main.dispatch task 82a6c374-7ce7-4cb8-aa2a-dcd256d24f8b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:04:45,209 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:04:47,918 DEBUG awx.main.scheduler Starting Scheduler
2021-05-16 11:04:48,863 DEBUG awx.main.dispatch task 82a6c374-7ce7-4cb8-aa2a-dcd256d24f8b is finished
2021-05-16 11:04:49,215 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(556355d3-2b7a-4b30-a455-d49b09c7d2da, queue=awx_private_queue)
[2021-05-16 11:04:49,215: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(556355d3-2b7a-4b30-a455-d49b09c7d2da, queue=awx_private_queue)
2021-05-16 11:04:49,235 DEBUG awx.main.dispatch delivered 556355d3-2b7a-4b30-a455-d49b09c7d2da to worker[213] qsize 0
2021-05-16 11:04:49,239 DEBUG awx.main.dispatch task 556355d3-2b7a-4b30-a455-d49b09c7d2da starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:04:49,425 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:04:49,468 DEBUG awx.main.tasks Last scheduler run was: 2021-05-16 11:04:30.749951+00:00
2021-05-16 11:04:49,514 DEBUG awx.main.dispatch task 556355d3-2b7a-4b30-a455-d49b09c7d2da is finished
RESULT 2
OKREADY
2021-05-16 11:05:05,198 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(12db9a01-baa7-4aed-abc6-7913d7d6c0d7, queue=awx_private_queue)
[2021-05-16 11:05:05,198: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(12db9a01-baa7-4aed-abc6-7913d7d6c0d7, queue=awx_private_queue)
2021-05-16 11:05:05,215 DEBUG awx.main.dispatch delivered 12db9a01-baa7-4aed-abc6-7913d7d6c0d7 to worker[216] qsize 0
2021-05-16 11:05:05,220 DEBUG awx.main.dispatch task 12db9a01-baa7-4aed-abc6-7913d7d6c0d7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:05:05,224 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:05:05,237 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:05:05,245 DEBUG awx.main.dispatch task 12db9a01-baa7-4aed-abc6-7913d7d6c0d7 is finished
2021-05-16 11:05:10,715 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(4120478f-108d-45da-bbf7-a7c584883e4b, queue=dev-awx-0)
[2021-05-16 11:05:10,715: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(4120478f-108d-45da-bbf7-a7c584883e4b, queue=dev-awx-0)
2021-05-16 11:05:10,729 DEBUG awx.main.dispatch delivered 4120478f-108d-45da-bbf7-a7c584883e4b to worker[211] qsize 0
2021-05-16 11:05:10,733 DEBUG awx.main.dispatch task 4120478f-108d-45da-bbf7-a7c584883e4b starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-05-16 11:05:10,737 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-05-16 11:05:10,755 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 470, in awx_isolated_heartbeat
for isolated_instance in isolated_instance_qs:
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-16 11:05:10,764 DEBUG awx.main.dispatch task 4120478f-108d-45da-bbf7-a7c584883e4b is finished
2021-05-16 11:05:19,212 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ea572991-9c7e-43ce-99b2-bbeb7824fd8f, queue=awx_private_queue)
[2021-05-16 11:05:19,212: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ea572991-9c7e-43ce-99b2-bbeb7824fd8f, queue=awx_private_queue)
2021-05-16 11:05:19,243 DEBUG awx.main.dispatch delivered ea572991-9c7e-43ce-99b2-bbeb7824fd8f to worker[233] qsize 0
2021-05-16 11:05:19,281 DEBUG awx.main.dispatch task ea572991-9c7e-43ce-99b2-bbeb7824fd8f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:05:19,313 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:05:19,316 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:05:19,323 DEBUG awx.main.dispatch task ea572991-9c7e-43ce-99b2-bbeb7824fd8f is finished
2021-05-16 11:05:25,136 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c9723f4c-db50-4f1c-aa42-ce510af8dc8f, queue=dev-awx-0)
[2021-05-16 11:05:25,136: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c9723f4c-db50-4f1c-aa42-ce510af8dc8f, queue=dev-awx-0)
2021-05-16 11:05:25,159 WARNING awx.main.dispatch scaling down worker pid:202
2021-05-16 11:05:25,164 WARNING awx.main.dispatch scaling down worker pid:211
2021-05-16 11:05:25,167 WARNING awx.main.dispatch scaling down worker pid:213
2021-05-16 11:05:25,170 WARNING awx.main.dispatch scaling down worker pid:216
2021-05-16 11:05:25,173 WARNING awx.main.dispatch scaling down worker pid:218
2021-05-16 11:05:25,168 WARNING awx.main.dispatch worker exiting gracefully pid:211
2021-05-16 11:05:25,168 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(852d85b5-bd72-4ea6-8883-b44447a807ea, queue=dev-awx-0)
[2021-05-16 11:05:25,168: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(852d85b5-bd72-4ea6-8883-b44447a807ea, queue=dev-awx-0)
2021-05-16 11:05:25,182 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-16 11:05:25,174 WARNING awx.main.dispatch worker exiting gracefully pid:216
2021-05-16 11:05:25,201 DEBUG awx.main.dispatch delivered 852d85b5-bd72-4ea6-8883-b44447a807ea to worker[229] qsize 0
2021-05-16 11:05:25,203 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0e642058-4993-4a5c-8b0e-af7066c0f696, queue=awx_private_queue)
[2021-05-16 11:05:25,203: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0e642058-4993-4a5c-8b0e-af7066c0f696, queue=awx_private_queue)
2021-05-16 11:05:25,223 DEBUG awx.main.dispatch delivered 0e642058-4993-4a5c-8b0e-af7066c0f696 to worker[231] qsize 0
2021-05-16 11:05:25,212 DEBUG awx.main.dispatch task 852d85b5-bd72-4ea6-8883-b44447a807ea starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:05:25,251 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140193118619200_1" does not exist
2021-05-16 11:05:25,257 DEBUG awx.main.dispatch task 852d85b5-bd72-4ea6-8883-b44447a807ea is finished
2021-05-16 11:05:25,231 DEBUG awx.main.dispatch task 0e642058-4993-4a5c-8b0e-af7066c0f696 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:05:25,264 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:05:25,266 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:05:25,270 DEBUG awx.main.dispatch task 0e642058-4993-4a5c-8b0e-af7066c0f696 is finished
2021-05-16 11:05:25,165 WARNING awx.main.dispatch worker exiting gracefully pid:202
2021-05-16 11:05:25,171 WARNING awx.main.dispatch worker exiting gracefully pid:213
2021-05-16 11:05:25,178 WARNING awx.main.dispatch worker exiting gracefully pid:218
2021-05-16 11:05:45,227 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ca6ef419-3ab4-4f03-851d-a042b2b66639, queue=awx_private_queue)
[2021-05-16 11:05:45,227: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ca6ef419-3ab4-4f03-851d-a042b2b66639, queue=awx_private_queue)
2021-05-16 11:05:45,259 DEBUG awx.main.dispatch delivered ca6ef419-3ab4-4f03-851d-a042b2b66639 to worker[220] qsize 0
2021-05-16 11:05:45,267 DEBUG awx.main.dispatch task ca6ef419-3ab4-4f03-851d-a042b2b66639 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:05:45,272 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:05:45,286 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:05:45,292 DEBUG awx.main.dispatch task ca6ef419-3ab4-4f03-851d-a042b2b66639 is finished
2021-05-16 11:05:49,212 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b2da8655-6159-4f71-be9a-19278dfc7fca, queue=awx_private_queue)
[2021-05-16 11:05:49,212: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b2da8655-6159-4f71-be9a-19278dfc7fca, queue=awx_private_queue)
2021-05-16 11:05:49,232 DEBUG awx.main.dispatch delivered b2da8655-6159-4f71-be9a-19278dfc7fca to worker[231] qsize 0
2021-05-16 11:05:49,238 DEBUG awx.main.dispatch task b2da8655-6159-4f71-be9a-19278dfc7fca starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:05:49,252 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:05:49,262 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:05:49,267 DEBUG awx.main.dispatch task b2da8655-6159-4f71-be9a-19278dfc7fca is finished
RESULT 2
OKREADY
2021-05-16 11:06:05,227 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4ea929fa-e1b6-4cc1-a99a-372d952ea272, queue=awx_private_queue)
[2021-05-16 11:06:05,227: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4ea929fa-e1b6-4cc1-a99a-372d952ea272, queue=awx_private_queue)
2021-05-16 11:06:05,252 DEBUG awx.main.dispatch delivered 4ea929fa-e1b6-4cc1-a99a-372d952ea272 to worker[229] qsize 0
2021-05-16 11:06:05,260 DEBUG awx.main.dispatch task 4ea929fa-e1b6-4cc1-a99a-372d952ea272 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:06:05,264 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:06:05,282 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:06:05,285 DEBUG awx.main.dispatch task 4ea929fa-e1b6-4cc1-a99a-372d952ea272 is finished
2021-05-16 11:06:19,217 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(58a5452b-9957-453a-a0b7-d1e8f230cba9, queue=awx_private_queue)
[2021-05-16 11:06:19,217: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(58a5452b-9957-453a-a0b7-d1e8f230cba9, queue=awx_private_queue)
2021-05-16 11:06:19,231 DEBUG awx.main.dispatch delivered 58a5452b-9957-453a-a0b7-d1e8f230cba9 to worker[233] qsize 0
2021-05-16 11:06:19,240 DEBUG awx.main.dispatch task 58a5452b-9957-453a-a0b7-d1e8f230cba9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:06:19,253 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:06:19,257 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:06:19,259 DEBUG awx.main.dispatch task 58a5452b-9957-453a-a0b7-d1e8f230cba9 is finished
2021-05-16 11:06:25,139 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(89e0050f-866b-42b5-8b74-24aee97cafd2, queue=dev-awx-0)
[2021-05-16 11:06:25,139: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(89e0050f-866b-42b5-8b74-24aee97cafd2, queue=dev-awx-0)
2021-05-16 11:06:25,167 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(6643451c-98ed-49b6-88bb-a699350620af, queue=dev-awx-0)
[2021-05-16 11:06:25,167: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(6643451c-98ed-49b6-88bb-a699350620af, queue=dev-awx-0)
2021-05-16 11:06:25,169 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-16 11:06:25,183 DEBUG awx.main.dispatch delivered 6643451c-98ed-49b6-88bb-a699350620af to worker[220] qsize 0
2021-05-16 11:06:25,188 DEBUG awx.main.dispatch task 6643451c-98ed-49b6-88bb-a699350620af starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:06:25,201 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140193118619200_2" does not exist
2021-05-16 11:06:25,204 DEBUG awx.main.dispatch task 6643451c-98ed-49b6-88bb-a699350620af is finished
2021-05-16 11:06:25,226 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(003916c4-2d54-4a9f-84e4-3250236fbb2f, queue=awx_private_queue)
[2021-05-16 11:06:25,226: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(003916c4-2d54-4a9f-84e4-3250236fbb2f, queue=awx_private_queue)
2021-05-16 11:06:25,247 DEBUG awx.main.dispatch delivered 003916c4-2d54-4a9f-84e4-3250236fbb2f to worker[231] qsize 0
2021-05-16 11:06:25,251 DEBUG awx.main.dispatch task 003916c4-2d54-4a9f-84e4-3250236fbb2f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:06:25,254 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:06:25,262 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:06:25,266 DEBUG awx.main.dispatch task 003916c4-2d54-4a9f-84e4-3250236fbb2f is finished
2021-05-16 11:06:45,236 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6a87ce56-3dc5-4364-bb5f-b14f46b05d0e, queue=awx_private_queue)
[2021-05-16 11:06:45,236: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6a87ce56-3dc5-4364-bb5f-b14f46b05d0e, queue=awx_private_queue)
2021-05-16 11:06:45,267 DEBUG awx.main.dispatch delivered 6a87ce56-3dc5-4364-bb5f-b14f46b05d0e to worker[233] qsize 0
2021-05-16 11:06:45,286 DEBUG awx.main.dispatch task 6a87ce56-3dc5-4364-bb5f-b14f46b05d0e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:06:45,304 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:06:45,305 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:06:45,310 DEBUG awx.main.dispatch task 6a87ce56-3dc5-4364-bb5f-b14f46b05d0e is finished
2021-05-16 11:06:49,217 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3acd7586-53ec-4e8a-946e-80fffd00e20d, queue=awx_private_queue)
[2021-05-16 11:06:49,217: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3acd7586-53ec-4e8a-946e-80fffd00e20d, queue=awx_private_queue)
2021-05-16 11:06:49,238 DEBUG awx.main.dispatch delivered 3acd7586-53ec-4e8a-946e-80fffd00e20d to worker[229] qsize 0
2021-05-16 11:06:49,247 DEBUG awx.main.dispatch task 3acd7586-53ec-4e8a-946e-80fffd00e20d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:06:49,271 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:06:49,274 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:06:49,280 DEBUG awx.main.dispatch task 3acd7586-53ec-4e8a-946e-80fffd00e20d is finished
RESULT 2
OKREADY
2021-05-16 11:07:05,252 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d5818818-c9e9-495e-a1f1-090708c007d0, queue=awx_private_queue)
[2021-05-16 11:07:05,252: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d5818818-c9e9-495e-a1f1-090708c007d0, queue=awx_private_queue)
2021-05-16 11:07:05,301 DEBUG awx.main.dispatch delivered d5818818-c9e9-495e-a1f1-090708c007d0 to worker[220] qsize 0
2021-05-16 11:07:05,315 DEBUG awx.main.dispatch task d5818818-c9e9-495e-a1f1-090708c007d0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:07:05,347 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:07:05,349 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:07:05,356 DEBUG awx.main.dispatch task d5818818-c9e9-495e-a1f1-090708c007d0 is finished
2021-05-16 11:07:19,220 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f5badfcf-4df7-4fb7-9294-335628889064, queue=awx_private_queue)
[2021-05-16 11:07:19,220: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f5badfcf-4df7-4fb7-9294-335628889064, queue=awx_private_queue)
2021-05-16 11:07:19,244 DEBUG awx.main.dispatch delivered f5badfcf-4df7-4fb7-9294-335628889064 to worker[220] qsize 0
2021-05-16 11:07:19,254 DEBUG awx.main.dispatch task f5badfcf-4df7-4fb7-9294-335628889064 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:07:19,272 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:07:19,276 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:07:19,285 DEBUG awx.main.dispatch task f5badfcf-4df7-4fb7-9294-335628889064 is finished
2021-05-16 11:07:25,135 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(4f574943-e247-45eb-ad08-12c5a58652d4, queue=dev-awx-0)
[2021-05-16 11:07:25,135: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(4f574943-e247-45eb-ad08-12c5a58652d4, queue=dev-awx-0)
2021-05-16 11:07:25,165 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-16 11:07:25,167 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2075bb76-75a8-4589-a0d7-113afeb28701, queue=dev-awx-0)
[2021-05-16 11:07:25,167: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2075bb76-75a8-4589-a0d7-113afeb28701, queue=dev-awx-0)
2021-05-16 11:07:25,193 DEBUG awx.main.dispatch delivered 2075bb76-75a8-4589-a0d7-113afeb28701 to worker[233] qsize 0
2021-05-16 11:07:25,202 DEBUG awx.main.dispatch task 2075bb76-75a8-4589-a0d7-113afeb28701 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:07:25,220 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140193118619200_1" does not exist
2021-05-16 11:07:25,224 DEBUG awx.main.dispatch task 2075bb76-75a8-4589-a0d7-113afeb28701 is finished
2021-05-16 11:07:25,243 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c3f4e25d-e0ba-4a39-b996-eb31a565ae53, queue=awx_private_queue)
[2021-05-16 11:07:25,243: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c3f4e25d-e0ba-4a39-b996-eb31a565ae53, queue=awx_private_queue)
2021-05-16 11:07:25,272 DEBUG awx.main.dispatch delivered c3f4e25d-e0ba-4a39-b996-eb31a565ae53 to worker[229] qsize 0
2021-05-16 11:07:25,279 DEBUG awx.main.dispatch task c3f4e25d-e0ba-4a39-b996-eb31a565ae53 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:07:25,296 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:07:25,298 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:07:25,304 DEBUG awx.main.dispatch task c3f4e25d-e0ba-4a39-b996-eb31a565ae53 is finished
2021-05-16 11:07:45,246 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6dbaa90f-1618-40f1-8539-5acfca81f9bd, queue=awx_private_queue)
[2021-05-16 11:07:45,246: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6dbaa90f-1618-40f1-8539-5acfca81f9bd, queue=awx_private_queue)
2021-05-16 11:07:45,268 DEBUG awx.main.dispatch delivered 6dbaa90f-1618-40f1-8539-5acfca81f9bd to worker[220] qsize 0
2021-05-16 11:07:45,277 DEBUG awx.main.dispatch task 6dbaa90f-1618-40f1-8539-5acfca81f9bd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:07:45,289 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:07:45,296 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:07:45,300 DEBUG awx.main.dispatch task 6dbaa90f-1618-40f1-8539-5acfca81f9bd is finished
2021-05-16 11:07:49,225 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9dfe6b77-ebc8-45a1-a073-6633a257b625, queue=awx_private_queue)
[2021-05-16 11:07:49,225: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9dfe6b77-ebc8-45a1-a073-6633a257b625, queue=awx_private_queue)
2021-05-16 11:07:49,257 DEBUG awx.main.dispatch delivered 9dfe6b77-ebc8-45a1-a073-6633a257b625 to worker[231] qsize 0
2021-05-16 11:07:49,267 DEBUG awx.main.dispatch task 9dfe6b77-ebc8-45a1-a073-6633a257b625 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:07:49,285 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:07:49,287 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:07:49,292 DEBUG awx.main.dispatch task 9dfe6b77-ebc8-45a1-a073-6633a257b625 is finished
RESULT 2
OKREADY
2021-05-16 11:08:05,250 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(09a13e7d-f4f8-40dc-97cc-b19c8c426f72, queue=awx_private_queue)
[2021-05-16 11:08:05,250: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(09a13e7d-f4f8-40dc-97cc-b19c8c426f72, queue=awx_private_queue)
2021-05-16 11:08:05,278 DEBUG awx.main.dispatch delivered 09a13e7d-f4f8-40dc-97cc-b19c8c426f72 to worker[220] qsize 0
2021-05-16 11:08:05,293 DEBUG awx.main.dispatch task 09a13e7d-f4f8-40dc-97cc-b19c8c426f72 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:08:05,313 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:08:05,316 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:08:05,322 DEBUG awx.main.dispatch task 09a13e7d-f4f8-40dc-97cc-b19c8c426f72 is finished
2021-05-16 11:08:19,222 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1d820594-e498-4c7b-b397-353219afa524, queue=awx_private_queue)
[2021-05-16 11:08:19,222: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1d820594-e498-4c7b-b397-353219afa524, queue=awx_private_queue)
2021-05-16 11:08:19,247 DEBUG awx.main.dispatch delivered 1d820594-e498-4c7b-b397-353219afa524 to worker[233] qsize 0
2021-05-16 11:08:19,263 DEBUG awx.main.dispatch task 1d820594-e498-4c7b-b397-353219afa524 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:08:19,282 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:08:19,285 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:08:19,290 DEBUG awx.main.dispatch task 1d820594-e498-4c7b-b397-353219afa524 is finished
2021-05-16 11:08:25,136 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(99971c02-c6a2-47f9-8a4a-d2bd02aa2751, queue=dev-awx-0)
[2021-05-16 11:08:25,136: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(99971c02-c6a2-47f9-8a4a-d2bd02aa2751, queue=dev-awx-0)
2021-05-16 11:08:25,168 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f4b37f60-7487-4a22-aa6b-59e94ed949d1, queue=dev-awx-0)
[2021-05-16 11:08:25,168: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f4b37f60-7487-4a22-aa6b-59e94ed949d1, queue=dev-awx-0)
2021-05-16 11:08:25,172 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-16 11:08:25,192 DEBUG awx.main.dispatch delivered f4b37f60-7487-4a22-aa6b-59e94ed949d1 to worker[229] qsize 0
2021-05-16 11:08:25,200 DEBUG awx.main.dispatch task f4b37f60-7487-4a22-aa6b-59e94ed949d1 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:08:25,228 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140193118619200_2" does not exist
2021-05-16 11:08:25,234 DEBUG awx.main.dispatch task f4b37f60-7487-4a22-aa6b-59e94ed949d1 is finished
2021-05-16 11:08:25,249 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(69b9ba32-7b10-49e4-be74-f38184c81a80, queue=awx_private_queue)
[2021-05-16 11:08:25,249: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(69b9ba32-7b10-49e4-be74-f38184c81a80, queue=awx_private_queue)
2021-05-16 11:08:25,271 DEBUG awx.main.dispatch delivered 69b9ba32-7b10-49e4-be74-f38184c81a80 to worker[233] qsize 0
2021-05-16 11:08:25,279 DEBUG awx.main.dispatch task 69b9ba32-7b10-49e4-be74-f38184c81a80 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:08:25,290 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:08:25,291 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:08:25,296 DEBUG awx.main.dispatch task 69b9ba32-7b10-49e4-be74-f38184c81a80 is finished
2021-05-16 11:08:45,258 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f8c87968-1517-49a3-9c26-9762978b4f7e, queue=awx_private_queue)
[2021-05-16 11:08:45,258: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f8c87968-1517-49a3-9c26-9762978b4f7e, queue=awx_private_queue)
2021-05-16 11:08:45,283 DEBUG awx.main.dispatch delivered f8c87968-1517-49a3-9c26-9762978b4f7e to worker[231] qsize 0
2021-05-16 11:08:45,291 DEBUG awx.main.dispatch task f8c87968-1517-49a3-9c26-9762978b4f7e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:08:45,309 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:08:45,310 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:08:45,315 DEBUG awx.main.dispatch task f8c87968-1517-49a3-9c26-9762978b4f7e is finished
2021-05-16 11:08:49,222 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0723ed36-6aaf-49de-85ce-2c4d4be18dbb, queue=awx_private_queue)
[2021-05-16 11:08:49,222: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0723ed36-6aaf-49de-85ce-2c4d4be18dbb, queue=awx_private_queue)
2021-05-16 11:08:49,253 DEBUG awx.main.dispatch delivered 0723ed36-6aaf-49de-85ce-2c4d4be18dbb to worker[233] qsize 0
2021-05-16 11:08:49,258 DEBUG awx.main.dispatch task 0723ed36-6aaf-49de-85ce-2c4d4be18dbb starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:08:49,270 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:08:49,272 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:08:49,275 DEBUG awx.main.dispatch task 0723ed36-6aaf-49de-85ce-2c4d4be18dbb is finished
RESULT 2
OKREADY
2021-05-16 11:09:05,273 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(83f77b7f-14ab-4209-b1f4-f65c240bd485, queue=awx_private_queue)
[2021-05-16 11:09:05,273: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(83f77b7f-14ab-4209-b1f4-f65c240bd485, queue=awx_private_queue)
2021-05-16 11:09:05,307 DEBUG awx.main.dispatch delivered 83f77b7f-14ab-4209-b1f4-f65c240bd485 to worker[229] qsize 0
2021-05-16 11:09:05,315 DEBUG awx.main.dispatch task 83f77b7f-14ab-4209-b1f4-f65c240bd485 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:09:05,326 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:09:05,328 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:09:05,332 DEBUG awx.main.dispatch task 83f77b7f-14ab-4209-b1f4-f65c240bd485 is finished
2021-05-16 11:09:19,230 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ad524f48-0219-4db8-be11-a12142a4af46, queue=awx_private_queue)
[2021-05-16 11:09:19,230: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ad524f48-0219-4db8-be11-a12142a4af46, queue=awx_private_queue)
2021-05-16 11:09:19,266 DEBUG awx.main.dispatch delivered ad524f48-0219-4db8-be11-a12142a4af46 to worker[220] qsize 0
2021-05-16 11:09:19,274 DEBUG awx.main.dispatch task ad524f48-0219-4db8-be11-a12142a4af46 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:09:19,292 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:09:19,294 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:09:19,299 DEBUG awx.main.dispatch task ad524f48-0219-4db8-be11-a12142a4af46 is finished
2021-05-16 11:09:25,135 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(cb516e2e-079e-4aa2-bcee-cb772d449fd8, queue=dev-awx-0)
[2021-05-16 11:09:25,135: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(cb516e2e-079e-4aa2-bcee-cb772d449fd8, queue=dev-awx-0)
2021-05-16 11:09:25,168 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(44dc38b5-d314-4b62-b96e-fc21df0ed840, queue=dev-awx-0)
[2021-05-16 11:09:25,168: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(44dc38b5-d314-4b62-b96e-fc21df0ed840, queue=dev-awx-0)
2021-05-16 11:09:25,168 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-16 11:09:25,188 DEBUG awx.main.dispatch delivered 44dc38b5-d314-4b62-b96e-fc21df0ed840 to worker[229] qsize 0
2021-05-16 11:09:25,210 DEBUG awx.main.dispatch task 44dc38b5-d314-4b62-b96e-fc21df0ed840 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:09:25,225 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140193118619200_3" does not exist
2021-05-16 11:09:25,229 DEBUG awx.main.dispatch task 44dc38b5-d314-4b62-b96e-fc21df0ed840 is finished
2021-05-16 11:09:25,266 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(960fa11b-0de4-4b17-a62c-9adc7616840e, queue=awx_private_queue)
[2021-05-16 11:09:25,266: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(960fa11b-0de4-4b17-a62c-9adc7616840e, queue=awx_private_queue)
2021-05-16 11:09:25,293 DEBUG awx.main.dispatch delivered 960fa11b-0de4-4b17-a62c-9adc7616840e to worker[229] qsize 0
2021-05-16 11:09:25,301 DEBUG awx.main.dispatch task 960fa11b-0de4-4b17-a62c-9adc7616840e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:09:25,310 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:09:25,311 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:09:25,315 DEBUG awx.main.dispatch task 960fa11b-0de4-4b17-a62c-9adc7616840e is finished
2021-05-16 11:09:45,276 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(adae6ca3-9967-4d71-aab7-eed2e5ddbadb, queue=awx_private_queue)
[2021-05-16 11:09:45,276: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(adae6ca3-9967-4d71-aab7-eed2e5ddbadb, queue=awx_private_queue)
2021-05-16 11:09:45,311 DEBUG awx.main.dispatch delivered adae6ca3-9967-4d71-aab7-eed2e5ddbadb to worker[220] qsize 0
2021-05-16 11:09:45,322 DEBUG awx.main.dispatch task adae6ca3-9967-4d71-aab7-eed2e5ddbadb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:09:45,335 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:09:45,336 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:09:45,340 DEBUG awx.main.dispatch task adae6ca3-9967-4d71-aab7-eed2e5ddbadb is finished
2021-05-16 11:09:49,227 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(593f80a6-4426-4ecb-952b-9fdb8a43aab1, queue=awx_private_queue)
[2021-05-16 11:09:49,227: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(593f80a6-4426-4ecb-952b-9fdb8a43aab1, queue=awx_private_queue)
2021-05-16 11:09:49,260 DEBUG awx.main.dispatch delivered 593f80a6-4426-4ecb-952b-9fdb8a43aab1 to worker[220] qsize 0
2021-05-16 11:09:49,266 DEBUG awx.main.dispatch task 593f80a6-4426-4ecb-952b-9fdb8a43aab1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:09:49,277 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:09:49,279 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:09:49,283 DEBUG awx.main.dispatch task 593f80a6-4426-4ecb-952b-9fdb8a43aab1 is finished
RESULT 2
OKREADY
2021-05-16 11:10:05,277 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9ac47d27-a33d-4c37-abcc-b37e6782e29d, queue=awx_private_queue)
[2021-05-16 11:10:05,277: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9ac47d27-a33d-4c37-abcc-b37e6782e29d, queue=awx_private_queue)
2021-05-16 11:10:05,302 DEBUG awx.main.dispatch delivered 9ac47d27-a33d-4c37-abcc-b37e6782e29d to worker[220] qsize 0
2021-05-16 11:10:05,309 DEBUG awx.main.dispatch task 9ac47d27-a33d-4c37-abcc-b37e6782e29d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:10:05,321 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:10:05,322 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:10:05,327 DEBUG awx.main.dispatch task 9ac47d27-a33d-4c37-abcc-b37e6782e29d is finished
2021-05-16 11:10:19,233 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ec2936b6-c5cf-4062-87a2-723f85172f17, queue=awx_private_queue)
[2021-05-16 11:10:19,233: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ec2936b6-c5cf-4062-87a2-723f85172f17, queue=awx_private_queue)
2021-05-16 11:10:19,256 DEBUG awx.main.dispatch delivered ec2936b6-c5cf-4062-87a2-723f85172f17 to worker[233] qsize 0
2021-05-16 11:10:19,266 DEBUG awx.main.dispatch task ec2936b6-c5cf-4062-87a2-723f85172f17 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:10:19,288 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:10:19,291 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:10:19,296 DEBUG awx.main.dispatch task ec2936b6-c5cf-4062-87a2-723f85172f17 is finished
2021-05-16 11:10:25,136 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(2fcf841c-b22e-4307-a0cf-77bbfe8d228f, queue=dev-awx-0)
[2021-05-16 11:10:25,136: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(2fcf841c-b22e-4307-a0cf-77bbfe8d228f, queue=dev-awx-0)
2021-05-16 11:10:25,167 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-16 11:10:25,168 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5df7b521-2620-438d-9ebd-036665ffd2bc, queue=dev-awx-0)
[2021-05-16 11:10:25,168: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5df7b521-2620-438d-9ebd-036665ffd2bc, queue=dev-awx-0)
2021-05-16 11:10:25,189 DEBUG awx.main.dispatch delivered 5df7b521-2620-438d-9ebd-036665ffd2bc to worker[231] qsize 0
2021-05-16 11:10:25,200 DEBUG awx.main.dispatch task 5df7b521-2620-438d-9ebd-036665ffd2bc starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:10:25,227 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140193118619200_2" does not exist
2021-05-16 11:10:25,232 DEBUG awx.main.dispatch task 5df7b521-2620-438d-9ebd-036665ffd2bc is finished
2021-05-16 11:10:25,277 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(05350fbc-202a-43f0-921d-21004fbf7e47, queue=awx_private_queue)
[2021-05-16 11:10:25,277: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(05350fbc-202a-43f0-921d-21004fbf7e47, queue=awx_private_queue)
2021-05-16 11:10:25,300 DEBUG awx.main.dispatch delivered 05350fbc-202a-43f0-921d-21004fbf7e47 to worker[229] qsize 0
2021-05-16 11:10:25,314 DEBUG awx.main.dispatch task 05350fbc-202a-43f0-921d-21004fbf7e47 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:10:25,332 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:10:25,333 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:10:25,338 DEBUG awx.main.dispatch task 05350fbc-202a-43f0-921d-21004fbf7e47 is finished
2021-05-16 11:10:45,291 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a37df0ed-14d1-4dfb-b238-237aebab33f2, queue=awx_private_queue)
[2021-05-16 11:10:45,291: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a37df0ed-14d1-4dfb-b238-237aebab33f2, queue=awx_private_queue)
2021-05-16 11:10:45,329 DEBUG awx.main.dispatch delivered a37df0ed-14d1-4dfb-b238-237aebab33f2 to worker[231] qsize 0
2021-05-16 11:10:45,339 DEBUG awx.main.dispatch task a37df0ed-14d1-4dfb-b238-237aebab33f2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:10:45,349 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:10:45,350 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:10:45,355 DEBUG awx.main.dispatch task a37df0ed-14d1-4dfb-b238-237aebab33f2 is finished
2021-05-16 11:10:49,232 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0b7e2488-0adb-4fec-9d97-bdfad8012c6c, queue=awx_private_queue)
[2021-05-16 11:10:49,232: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0b7e2488-0adb-4fec-9d97-bdfad8012c6c, queue=awx_private_queue)
2021-05-16 11:10:49,255 DEBUG awx.main.dispatch delivered 0b7e2488-0adb-4fec-9d97-bdfad8012c6c to worker[229] qsize 0
2021-05-16 11:10:49,263 DEBUG awx.main.dispatch task 0b7e2488-0adb-4fec-9d97-bdfad8012c6c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:10:49,280 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:10:49,283 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:10:49,289 DEBUG awx.main.dispatch task 0b7e2488-0adb-4fec-9d97-bdfad8012c6c is finished
RESULT 2
OKREADY
2021-05-16 11:11:05,289 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d023ef78-2dd4-42f7-bdec-acad5da475bb, queue=awx_private_queue)
[2021-05-16 11:11:05,289: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d023ef78-2dd4-42f7-bdec-acad5da475bb, queue=awx_private_queue)
2021-05-16 11:11:05,314 DEBUG awx.main.dispatch delivered d023ef78-2dd4-42f7-bdec-acad5da475bb to worker[229] qsize 0
2021-05-16 11:11:05,325 DEBUG awx.main.dispatch task d023ef78-2dd4-42f7-bdec-acad5da475bb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:11:05,339 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:11:05,340 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:11:05,344 DEBUG awx.main.dispatch task d023ef78-2dd4-42f7-bdec-acad5da475bb is finished
2021-05-16 11:11:19,234 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2aade9b2-80d8-4c54-b0f4-405e246e48c1, queue=awx_private_queue)
[2021-05-16 11:11:19,234: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2aade9b2-80d8-4c54-b0f4-405e246e48c1, queue=awx_private_queue)
2021-05-16 11:11:19,258 DEBUG awx.main.dispatch delivered 2aade9b2-80d8-4c54-b0f4-405e246e48c1 to worker[233] qsize 0
2021-05-16 11:11:19,266 DEBUG awx.main.dispatch task 2aade9b2-80d8-4c54-b0f4-405e246e48c1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:11:19,284 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:11:19,288 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:11:19,299 DEBUG awx.main.dispatch task 2aade9b2-80d8-4c54-b0f4-405e246e48c1 is finished
2021-05-16 11:11:25,136 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(fa4d864b-9d9e-4ff6-bf14-3f2021dd3640, queue=dev-awx-0)
[2021-05-16 11:11:25,136: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(fa4d864b-9d9e-4ff6-bf14-3f2021dd3640, queue=dev-awx-0)
2021-05-16 11:11:25,168 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ebba976d-c081-4139-9e87-3f8bdf7fabe4, queue=dev-awx-0)
[2021-05-16 11:11:25,168: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ebba976d-c081-4139-9e87-3f8bdf7fabe4, queue=dev-awx-0)
2021-05-16 11:11:25,165 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-16 11:11:25,189 DEBUG awx.main.dispatch delivered ebba976d-c081-4139-9e87-3f8bdf7fabe4 to worker[231] qsize 0
2021-05-16 11:11:25,197 DEBUG awx.main.dispatch task ebba976d-c081-4139-9e87-3f8bdf7fabe4 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:11:25,215 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140193118619200_3" does not exist
2021-05-16 11:11:25,222 DEBUG awx.main.dispatch task ebba976d-c081-4139-9e87-3f8bdf7fabe4 is finished
2021-05-16 11:11:25,288 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(df5f1594-95a4-48a3-a649-b68a9889f17e, queue=awx_private_queue)
[2021-05-16 11:11:25,288: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(df5f1594-95a4-48a3-a649-b68a9889f17e, queue=awx_private_queue)
2021-05-16 11:11:25,315 DEBUG awx.main.dispatch delivered df5f1594-95a4-48a3-a649-b68a9889f17e to worker[233] qsize 0
2021-05-16 11:11:25,322 DEBUG awx.main.dispatch task df5f1594-95a4-48a3-a649-b68a9889f17e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:11:25,332 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:11:25,333 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:11:25,337 DEBUG awx.main.dispatch task df5f1594-95a4-48a3-a649-b68a9889f17e is finished
2021-05-16 11:11:45,301 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(46800dae-e099-4122-9b9b-854830bf31fc, queue=awx_private_queue)
[2021-05-16 11:11:45,301: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(46800dae-e099-4122-9b9b-854830bf31fc, queue=awx_private_queue)
2021-05-16 11:11:45,325 DEBUG awx.main.dispatch delivered 46800dae-e099-4122-9b9b-854830bf31fc to worker[233] qsize 0
2021-05-16 11:11:45,332 DEBUG awx.main.dispatch task 46800dae-e099-4122-9b9b-854830bf31fc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:11:45,346 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:11:45,349 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:11:45,354 DEBUG awx.main.dispatch task 46800dae-e099-4122-9b9b-854830bf31fc is finished
2021-05-16 11:11:49,237 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(85afacfa-7f10-4f54-bbdc-8c447f92f3b3, queue=awx_private_queue)
[2021-05-16 11:11:49,237: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(85afacfa-7f10-4f54-bbdc-8c447f92f3b3, queue=awx_private_queue)
2021-05-16 11:11:49,272 DEBUG awx.main.dispatch delivered 85afacfa-7f10-4f54-bbdc-8c447f92f3b3 to worker[233] qsize 0
2021-05-16 11:11:49,282 DEBUG awx.main.dispatch task 85afacfa-7f10-4f54-bbdc-8c447f92f3b3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:11:49,297 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:11:49,299 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:11:49,303 DEBUG awx.main.dispatch task 85afacfa-7f10-4f54-bbdc-8c447f92f3b3 is finished
RESULT 2
OKREADY
2021-05-16 11:12:05,306 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9522ce21-5208-486d-9804-9f01d269aa66, queue=awx_private_queue)
[2021-05-16 11:12:05,306: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9522ce21-5208-486d-9804-9f01d269aa66, queue=awx_private_queue)
2021-05-16 11:12:05,352 DEBUG awx.main.dispatch delivered 9522ce21-5208-486d-9804-9f01d269aa66 to worker[229] qsize 0
2021-05-16 11:12:05,367 DEBUG awx.main.dispatch task 9522ce21-5208-486d-9804-9f01d269aa66 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:12:05,393 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:12:05,395 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:12:05,402 DEBUG awx.main.dispatch task 9522ce21-5208-486d-9804-9f01d269aa66 is finished
2021-05-16 11:12:19,243 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b7444e2b-e1da-43c7-ba35-2dabb60c5870, queue=awx_private_queue)
[2021-05-16 11:12:19,243: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b7444e2b-e1da-43c7-ba35-2dabb60c5870, queue=awx_private_queue)
2021-05-16 11:12:19,278 DEBUG awx.main.dispatch delivered b7444e2b-e1da-43c7-ba35-2dabb60c5870 to worker[231] qsize 0
2021-05-16 11:12:19,297 DEBUG awx.main.dispatch task b7444e2b-e1da-43c7-ba35-2dabb60c5870 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:12:19,315 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:12:19,317 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:12:19,323 DEBUG awx.main.dispatch task b7444e2b-e1da-43c7-ba35-2dabb60c5870 is finished
2021-05-16 11:12:25,136 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(089087e0-5936-4275-8d9e-3f29b791cd6a, queue=dev-awx-0)
[2021-05-16 11:12:25,136: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(089087e0-5936-4275-8d9e-3f29b791cd6a, queue=dev-awx-0)
2021-05-16 11:12:25,165 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-16 11:12:25,170 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(93d470b8-86ca-46f2-8b3f-1c38e91c7461, queue=dev-awx-0)
[2021-05-16 11:12:25,170: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(93d470b8-86ca-46f2-8b3f-1c38e91c7461, queue=dev-awx-0)
2021-05-16 11:12:25,214 DEBUG awx.main.dispatch delivered 93d470b8-86ca-46f2-8b3f-1c38e91c7461 to worker[233] qsize 0
2021-05-16 11:12:25,221 DEBUG awx.main.dispatch task 93d470b8-86ca-46f2-8b3f-1c38e91c7461 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:12:25,241 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140193118619200_2" does not exist
2021-05-16 11:12:25,246 DEBUG awx.main.dispatch task 93d470b8-86ca-46f2-8b3f-1c38e91c7461 is finished
2021-05-16 11:12:25,301 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(26e92c6d-ca22-449c-a8cd-4e2080721e99, queue=awx_private_queue)
[2021-05-16 11:12:25,301: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(26e92c6d-ca22-449c-a8cd-4e2080721e99, queue=awx_private_queue)
2021-05-16 11:12:25,336 DEBUG awx.main.dispatch delivered 26e92c6d-ca22-449c-a8cd-4e2080721e99 to worker[233] qsize 0
2021-05-16 11:12:25,346 DEBUG awx.main.dispatch task 26e92c6d-ca22-449c-a8cd-4e2080721e99 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:12:25,355 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:12:25,356 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:12:25,359 DEBUG awx.main.dispatch task 26e92c6d-ca22-449c-a8cd-4e2080721e99 is finished
2021-05-16 11:12:45,306 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2f3f324e-3e1f-4abe-87e4-8cf25a746033, queue=awx_private_queue)
[2021-05-16 11:12:45,306: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2f3f324e-3e1f-4abe-87e4-8cf25a746033, queue=awx_private_queue)
2021-05-16 11:12:45,338 DEBUG awx.main.dispatch delivered 2f3f324e-3e1f-4abe-87e4-8cf25a746033 to worker[220] qsize 0
2021-05-16 11:12:45,349 DEBUG awx.main.dispatch task 2f3f324e-3e1f-4abe-87e4-8cf25a746033 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:12:45,369 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:12:45,371 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:12:45,376 DEBUG awx.main.dispatch task 2f3f324e-3e1f-4abe-87e4-8cf25a746033 is finished
2021-05-16 11:12:49,237 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a9e81f05-9f15-45d0-9b18-fb8484cd8671, queue=awx_private_queue)
[2021-05-16 11:12:49,237: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a9e81f05-9f15-45d0-9b18-fb8484cd8671, queue=awx_private_queue)
2021-05-16 11:12:49,261 DEBUG awx.main.dispatch delivered a9e81f05-9f15-45d0-9b18-fb8484cd8671 to worker[229] qsize 0
2021-05-16 11:12:49,268 DEBUG awx.main.dispatch task a9e81f05-9f15-45d0-9b18-fb8484cd8671 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:12:49,283 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:12:49,285 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:12:49,289 DEBUG awx.main.dispatch task a9e81f05-9f15-45d0-9b18-fb8484cd8671 is finished
RESULT 2
OKREADY
2021-05-16 11:13:05,306 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e4005256-7f7c-4655-8927-49a34707be73, queue=awx_private_queue)
[2021-05-16 11:13:05,306: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e4005256-7f7c-4655-8927-49a34707be73, queue=awx_private_queue)
2021-05-16 11:13:05,331 DEBUG awx.main.dispatch delivered e4005256-7f7c-4655-8927-49a34707be73 to worker[231] qsize 0
2021-05-16 11:13:05,340 DEBUG awx.main.dispatch task e4005256-7f7c-4655-8927-49a34707be73 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:13:05,358 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:13:05,360 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:13:05,364 DEBUG awx.main.dispatch task e4005256-7f7c-4655-8927-49a34707be73 is finished
2021-05-16 11:13:19,244 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a28b5fda-329e-4ac0-a529-7d5e76575276, queue=awx_private_queue)
[2021-05-16 11:13:19,244: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a28b5fda-329e-4ac0-a529-7d5e76575276, queue=awx_private_queue)
2021-05-16 11:13:19,281 DEBUG awx.main.dispatch delivered a28b5fda-329e-4ac0-a529-7d5e76575276 to worker[220] qsize 0
2021-05-16 11:13:19,292 DEBUG awx.main.dispatch task a28b5fda-329e-4ac0-a529-7d5e76575276 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:13:19,310 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:13:19,313 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:13:19,319 DEBUG awx.main.dispatch task a28b5fda-329e-4ac0-a529-7d5e76575276 is finished
2021-05-16 11:13:25,137 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(63e84495-95fc-42b1-8825-f1ebb0d649e1, queue=dev-awx-0)
[2021-05-16 11:13:25,137: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(63e84495-95fc-42b1-8825-f1ebb0d649e1, queue=dev-awx-0)
2021-05-16 11:13:25,169 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(10cdb305-bb2b-4258-ba08-2403624b4700, queue=dev-awx-0)
[2021-05-16 11:13:25,169: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(10cdb305-bb2b-4258-ba08-2403624b4700, queue=dev-awx-0)
2021-05-16 11:13:25,166 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-16 11:13:25,196 DEBUG awx.main.dispatch delivered 10cdb305-bb2b-4258-ba08-2403624b4700 to worker[231] qsize 0
2021-05-16 11:13:25,202 DEBUG awx.main.dispatch task 10cdb305-bb2b-4258-ba08-2403624b4700 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:13:25,216 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140193118619200_4" does not exist
2021-05-16 11:13:25,219 DEBUG awx.main.dispatch task 10cdb305-bb2b-4258-ba08-2403624b4700 is finished
2021-05-16 11:13:25,306 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3630e115-582c-4c45-967e-5889e967bf56, queue=awx_private_queue)
[2021-05-16 11:13:25,306: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3630e115-582c-4c45-967e-5889e967bf56, queue=awx_private_queue)
2021-05-16 11:13:25,329 DEBUG awx.main.dispatch delivered 3630e115-582c-4c45-967e-5889e967bf56 to worker[233] qsize 0
2021-05-16 11:13:25,336 DEBUG awx.main.dispatch task 3630e115-582c-4c45-967e-5889e967bf56 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:13:25,353 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:13:25,354 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:13:25,358 DEBUG awx.main.dispatch task 3630e115-582c-4c45-967e-5889e967bf56 is finished
2021-05-16 11:13:45,318 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(51492185-b071-4d49-bb4e-6dbe2055b027, queue=awx_private_queue)
[2021-05-16 11:13:45,318: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(51492185-b071-4d49-bb4e-6dbe2055b027, queue=awx_private_queue)
2021-05-16 11:13:45,348 DEBUG awx.main.dispatch delivered 51492185-b071-4d49-bb4e-6dbe2055b027 to worker[229] qsize 0
2021-05-16 11:13:45,362 DEBUG awx.main.dispatch task 51492185-b071-4d49-bb4e-6dbe2055b027 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:13:45,376 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:13:45,378 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:13:45,382 DEBUG awx.main.dispatch task 51492185-b071-4d49-bb4e-6dbe2055b027 is finished
2021-05-16 11:13:49,241 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(580f63a9-bb4a-4618-a0dd-a544c0323740, queue=awx_private_queue)
[2021-05-16 11:13:49,241: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(580f63a9-bb4a-4618-a0dd-a544c0323740, queue=awx_private_queue)
2021-05-16 11:13:49,266 DEBUG awx.main.dispatch delivered 580f63a9-bb4a-4618-a0dd-a544c0323740 to worker[229] qsize 0
2021-05-16 11:13:49,273 DEBUG awx.main.dispatch task 580f63a9-bb4a-4618-a0dd-a544c0323740 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:13:49,291 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:13:49,293 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:13:49,298 DEBUG awx.main.dispatch task 580f63a9-bb4a-4618-a0dd-a544c0323740 is finished
RESULT 2
OKREADY
2021-05-16 11:14:05,321 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(01980225-7db7-4a86-8af8-a8ffe394f184, queue=awx_private_queue)
[2021-05-16 11:14:05,321: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(01980225-7db7-4a86-8af8-a8ffe394f184, queue=awx_private_queue)
2021-05-16 11:14:05,349 DEBUG awx.main.dispatch delivered 01980225-7db7-4a86-8af8-a8ffe394f184 to worker[229] qsize 0
2021-05-16 11:14:05,358 DEBUG awx.main.dispatch task 01980225-7db7-4a86-8af8-a8ffe394f184 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:14:05,370 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:14:05,371 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:14:05,376 DEBUG awx.main.dispatch task 01980225-7db7-4a86-8af8-a8ffe394f184 is finished
2021-05-16 11:14:19,245 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d0faa91d-0832-4065-9c60-5104335da906, queue=awx_private_queue)
[2021-05-16 11:14:19,245: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d0faa91d-0832-4065-9c60-5104335da906, queue=awx_private_queue)
2021-05-16 11:14:19,268 DEBUG awx.main.dispatch delivered d0faa91d-0832-4065-9c60-5104335da906 to worker[233] qsize 0
2021-05-16 11:14:19,277 DEBUG awx.main.dispatch task d0faa91d-0832-4065-9c60-5104335da906 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:14:19,295 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:14:19,298 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:14:19,303 DEBUG awx.main.dispatch task d0faa91d-0832-4065-9c60-5104335da906 is finished
2021-05-16 11:14:25,140 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8e48a3e8-f1a1-4fa8-9ea6-05353a5dd50e, queue=dev-awx-0)
[2021-05-16 11:14:25,140: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8e48a3e8-f1a1-4fa8-9ea6-05353a5dd50e, queue=dev-awx-0)
2021-05-16 11:14:25,174 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(61b96cef-9b47-4ff7-a7cb-211608e6fb89, queue=dev-awx-0)
[2021-05-16 11:14:25,174: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(61b96cef-9b47-4ff7-a7cb-211608e6fb89, queue=dev-awx-0)
2021-05-16 11:14:25,178 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-16 11:14:25,203 DEBUG awx.main.dispatch delivered 61b96cef-9b47-4ff7-a7cb-211608e6fb89 to worker[233] qsize 0
2021-05-16 11:14:25,209 DEBUG awx.main.dispatch task 61b96cef-9b47-4ff7-a7cb-211608e6fb89 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:14:25,221 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140193118619200_3" does not exist
2021-05-16 11:14:25,225 DEBUG awx.main.dispatch task 61b96cef-9b47-4ff7-a7cb-211608e6fb89 is finished
2021-05-16 11:14:25,321 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(444d86b3-bfc0-4189-8b8c-abcaeddf65fc, queue=awx_private_queue)
[2021-05-16 11:14:25,321: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(444d86b3-bfc0-4189-8b8c-abcaeddf65fc, queue=awx_private_queue)
2021-05-16 11:14:25,357 DEBUG awx.main.dispatch delivered 444d86b3-bfc0-4189-8b8c-abcaeddf65fc to worker[220] qsize 0
2021-05-16 11:14:25,365 DEBUG awx.main.dispatch task 444d86b3-bfc0-4189-8b8c-abcaeddf65fc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:14:25,384 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:14:25,386 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:14:25,390 DEBUG awx.main.dispatch task 444d86b3-bfc0-4189-8b8c-abcaeddf65fc is finished
2021-05-16 11:14:45,337 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ffc09472-8e93-413a-92a5-f5e29ed331c7, queue=awx_private_queue)
[2021-05-16 11:14:45,337: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ffc09472-8e93-413a-92a5-f5e29ed331c7, queue=awx_private_queue)
2021-05-16 11:14:45,365 DEBUG awx.main.dispatch delivered ffc09472-8e93-413a-92a5-f5e29ed331c7 to worker[220] qsize 0
2021-05-16 11:14:45,377 DEBUG awx.main.dispatch task ffc09472-8e93-413a-92a5-f5e29ed331c7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:14:45,389 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:14:45,390 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:14:45,397 DEBUG awx.main.dispatch task ffc09472-8e93-413a-92a5-f5e29ed331c7 is finished
2021-05-16 11:14:49,245 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(31f711bd-bef8-4f90-a582-6c294f370c11, queue=awx_private_queue)
[2021-05-16 11:14:49,245: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(31f711bd-bef8-4f90-a582-6c294f370c11, queue=awx_private_queue)
2021-05-16 11:14:49,272 DEBUG awx.main.dispatch delivered 31f711bd-bef8-4f90-a582-6c294f370c11 to worker[233] qsize 0
2021-05-16 11:14:49,280 DEBUG awx.main.dispatch task 31f711bd-bef8-4f90-a582-6c294f370c11 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:14:49,300 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:14:49,302 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:14:49,307 DEBUG awx.main.dispatch task 31f711bd-bef8-4f90-a582-6c294f370c11 is finished
RESULT 2
OKREADY
2021-05-16 11:15:05,337 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6a511d0e-51e3-43ab-973f-684cec8a9cca, queue=awx_private_queue)
[2021-05-16 11:15:05,337: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6a511d0e-51e3-43ab-973f-684cec8a9cca, queue=awx_private_queue)
2021-05-16 11:15:05,361 DEBUG awx.main.dispatch delivered 6a511d0e-51e3-43ab-973f-684cec8a9cca to worker[220] qsize 0
2021-05-16 11:15:05,371 DEBUG awx.main.dispatch task 6a511d0e-51e3-43ab-973f-684cec8a9cca starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:15:05,384 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:15:05,385 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:15:05,391 DEBUG awx.main.dispatch task 6a511d0e-51e3-43ab-973f-684cec8a9cca is finished
2021-05-16 11:15:10,717 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(d2a07b74-04dd-4e5a-976a-71b2f999ee8f, queue=dev-awx-0)
[2021-05-16 11:15:10,717: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(d2a07b74-04dd-4e5a-976a-71b2f999ee8f, queue=dev-awx-0)
2021-05-16 11:15:10,741 DEBUG awx.main.dispatch delivered d2a07b74-04dd-4e5a-976a-71b2f999ee8f to worker[231] qsize 0
2021-05-16 11:15:10,753 DEBUG awx.main.dispatch task d2a07b74-04dd-4e5a-976a-71b2f999ee8f starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-05-16 11:15:10,779 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-05-16 11:15:10,792 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 470, in awx_isolated_heartbeat
for isolated_instance in isolated_instance_qs:
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-16 11:15:10,798 DEBUG awx.main.dispatch task d2a07b74-04dd-4e5a-976a-71b2f999ee8f is finished
2021-05-16 11:15:19,249 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d36c0d53-a1b0-4284-a99b-dbf298a32b51, queue=awx_private_queue)
[2021-05-16 11:15:19,249: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d36c0d53-a1b0-4284-a99b-dbf298a32b51, queue=awx_private_queue)
2021-05-16 11:15:19,271 DEBUG awx.main.dispatch delivered d36c0d53-a1b0-4284-a99b-dbf298a32b51 to worker[220] qsize 0
2021-05-16 11:15:19,279 DEBUG awx.main.dispatch task d36c0d53-a1b0-4284-a99b-dbf298a32b51 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:15:19,291 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:15:19,293 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:15:19,297 DEBUG awx.main.dispatch task d36c0d53-a1b0-4284-a99b-dbf298a32b51 is finished
2021-05-16 11:15:25,136 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(be348b3c-0701-467d-8427-528282ba97ba, queue=dev-awx-0)
[2021-05-16 11:15:25,136: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(be348b3c-0701-467d-8427-528282ba97ba, queue=dev-awx-0)
2021-05-16 11:15:25,165 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-16 11:15:25,173 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(9a91a142-9c94-4492-9149-f86325214451, queue=dev-awx-0)
[2021-05-16 11:15:25,173: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(9a91a142-9c94-4492-9149-f86325214451, queue=dev-awx-0)
2021-05-16 11:15:25,194 DEBUG awx.main.dispatch delivered 9a91a142-9c94-4492-9149-f86325214451 to worker[233] qsize 0
2021-05-16 11:15:25,202 DEBUG awx.main.dispatch task 9a91a142-9c94-4492-9149-f86325214451 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:15:25,216 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140193118619200_4" does not exist
2021-05-16 11:15:25,220 DEBUG awx.main.dispatch task 9a91a142-9c94-4492-9149-f86325214451 is finished
2021-05-16 11:15:25,336 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7d3f17e7-1f7d-4d11-930e-69c2a3e9cbf9, queue=awx_private_queue)
[2021-05-16 11:15:25,336: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7d3f17e7-1f7d-4d11-930e-69c2a3e9cbf9, queue=awx_private_queue)
2021-05-16 11:15:25,360 DEBUG awx.main.dispatch delivered 7d3f17e7-1f7d-4d11-930e-69c2a3e9cbf9 to worker[231] qsize 0
2021-05-16 11:15:25,364 DEBUG awx.main.dispatch task 7d3f17e7-1f7d-4d11-930e-69c2a3e9cbf9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:15:25,373 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:15:25,374 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:15:25,378 DEBUG awx.main.dispatch task 7d3f17e7-1f7d-4d11-930e-69c2a3e9cbf9 is finished
2021-05-16 11:15:45,338 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e49455f0-3f46-45c2-b598-6a8827a65e7b, queue=awx_private_queue)
[2021-05-16 11:15:45,338: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e49455f0-3f46-45c2-b598-6a8827a65e7b, queue=awx_private_queue)
2021-05-16 11:15:45,359 DEBUG awx.main.dispatch delivered e49455f0-3f46-45c2-b598-6a8827a65e7b to worker[220] qsize 0
2021-05-16 11:15:45,365 DEBUG awx.main.dispatch task e49455f0-3f46-45c2-b598-6a8827a65e7b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:15:45,378 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:15:45,379 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:15:45,384 DEBUG awx.main.dispatch task e49455f0-3f46-45c2-b598-6a8827a65e7b is finished
2021-05-16 11:15:49,245 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(28c28449-496d-453c-aa46-ee008be8486c, queue=awx_private_queue)
[2021-05-16 11:15:49,245: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(28c28449-496d-453c-aa46-ee008be8486c, queue=awx_private_queue)
2021-05-16 11:15:49,274 DEBUG awx.main.dispatch delivered 28c28449-496d-453c-aa46-ee008be8486c to worker[233] qsize 0
2021-05-16 11:15:49,284 DEBUG awx.main.dispatch task 28c28449-496d-453c-aa46-ee008be8486c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:15:49,298 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:15:49,301 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:15:49,305 DEBUG awx.main.dispatch task 28c28449-496d-453c-aa46-ee008be8486c is finished
RESULT 2
OKREADY
2021-05-16 11:16:05,347 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f8db4ba2-2fce-413a-b4b4-8ddc4e07312d, queue=awx_private_queue)
[2021-05-16 11:16:05,347: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f8db4ba2-2fce-413a-b4b4-8ddc4e07312d, queue=awx_private_queue)
2021-05-16 11:16:05,380 DEBUG awx.main.dispatch delivered f8db4ba2-2fce-413a-b4b4-8ddc4e07312d to worker[220] qsize 0
2021-05-16 11:16:05,388 DEBUG awx.main.dispatch task f8db4ba2-2fce-413a-b4b4-8ddc4e07312d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:16:05,404 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:16:05,406 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:16:05,409 DEBUG awx.main.dispatch task f8db4ba2-2fce-413a-b4b4-8ddc4e07312d is finished
2021-05-16 11:16:19,248 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(eaba8805-9b9e-4b87-973d-92fa7cf03996, queue=awx_private_queue)
[2021-05-16 11:16:19,248: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(eaba8805-9b9e-4b87-973d-92fa7cf03996, queue=awx_private_queue)
2021-05-16 11:16:19,284 DEBUG awx.main.dispatch delivered eaba8805-9b9e-4b87-973d-92fa7cf03996 to worker[231] qsize 0
2021-05-16 11:16:19,295 DEBUG awx.main.dispatch task eaba8805-9b9e-4b87-973d-92fa7cf03996 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:16:19,319 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:16:19,321 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:16:19,326 DEBUG awx.main.dispatch task eaba8805-9b9e-4b87-973d-92fa7cf03996 is finished
2021-05-16 11:16:25,136 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(65b30b70-be34-4ba2-bca0-daa8fb1bd2cf, queue=dev-awx-0)
[2021-05-16 11:16:25,136: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(65b30b70-be34-4ba2-bca0-daa8fb1bd2cf, queue=dev-awx-0)
2021-05-16 11:16:25,167 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-16 11:16:25,174 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2885e2c4-06b5-443b-9de4-71b1b1738296, queue=dev-awx-0)
[2021-05-16 11:16:25,174: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2885e2c4-06b5-443b-9de4-71b1b1738296, queue=dev-awx-0)
2021-05-16 11:16:25,195 DEBUG awx.main.dispatch delivered 2885e2c4-06b5-443b-9de4-71b1b1738296 to worker[220] qsize 0
2021-05-16 11:16:25,201 DEBUG awx.main.dispatch task 2885e2c4-06b5-443b-9de4-71b1b1738296 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:16:25,218 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140193118619200_3" does not exist
2021-05-16 11:16:25,224 DEBUG awx.main.dispatch task 2885e2c4-06b5-443b-9de4-71b1b1738296 is finished
2021-05-16 11:16:25,340 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(41af0d63-bc3f-431b-9405-e6e481fce8d2, queue=awx_private_queue)
[2021-05-16 11:16:25,340: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(41af0d63-bc3f-431b-9405-e6e481fce8d2, queue=awx_private_queue)
2021-05-16 11:16:25,363 DEBUG awx.main.dispatch delivered 41af0d63-bc3f-431b-9405-e6e481fce8d2 to worker[231] qsize 0
2021-05-16 11:16:25,369 DEBUG awx.main.dispatch task 41af0d63-bc3f-431b-9405-e6e481fce8d2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:16:25,383 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:16:25,384 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:16:25,390 DEBUG awx.main.dispatch task 41af0d63-bc3f-431b-9405-e6e481fce8d2 is finished
2021-05-16 11:16:45,341 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ee5f8e98-ef9f-45b8-80da-15ed4b29a069, queue=awx_private_queue)
[2021-05-16 11:16:45,341: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ee5f8e98-ef9f-45b8-80da-15ed4b29a069, queue=awx_private_queue)
2021-05-16 11:16:45,368 DEBUG awx.main.dispatch delivered ee5f8e98-ef9f-45b8-80da-15ed4b29a069 to worker[233] qsize 0
2021-05-16 11:16:45,384 DEBUG awx.main.dispatch task ee5f8e98-ef9f-45b8-80da-15ed4b29a069 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:16:45,399 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:16:45,400 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:16:45,405 DEBUG awx.main.dispatch task ee5f8e98-ef9f-45b8-80da-15ed4b29a069 is finished
2021-05-16 11:16:49,245 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(35e994b0-4358-47d2-8720-3a960718bac2, queue=awx_private_queue)
[2021-05-16 11:16:49,245: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(35e994b0-4358-47d2-8720-3a960718bac2, queue=awx_private_queue)
2021-05-16 11:16:49,274 DEBUG awx.main.dispatch delivered 35e994b0-4358-47d2-8720-3a960718bac2 to worker[233] qsize 0
2021-05-16 11:16:49,280 DEBUG awx.main.dispatch task 35e994b0-4358-47d2-8720-3a960718bac2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:16:49,289 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:16:49,292 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:16:49,302 DEBUG awx.main.dispatch task 35e994b0-4358-47d2-8720-3a960718bac2 is finished
RESULT 2
OKREADY
2021-05-16 11:17:05,351 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(929d3edf-f11c-4c78-87dc-40b6c87919b6, queue=awx_private_queue)
[2021-05-16 11:17:05,351: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(929d3edf-f11c-4c78-87dc-40b6c87919b6, queue=awx_private_queue)
2021-05-16 11:17:05,379 DEBUG awx.main.dispatch delivered 929d3edf-f11c-4c78-87dc-40b6c87919b6 to worker[231] qsize 0
2021-05-16 11:17:05,391 DEBUG awx.main.dispatch task 929d3edf-f11c-4c78-87dc-40b6c87919b6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:17:05,410 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:17:05,411 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:17:05,416 DEBUG awx.main.dispatch task 929d3edf-f11c-4c78-87dc-40b6c87919b6 is finished
2021-05-16 11:17:19,247 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(28b0a3e8-b532-430d-8f48-80574b497eb2, queue=awx_private_queue)
[2021-05-16 11:17:19,247: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(28b0a3e8-b532-430d-8f48-80574b497eb2, queue=awx_private_queue)
2021-05-16 11:17:19,268 DEBUG awx.main.dispatch delivered 28b0a3e8-b532-430d-8f48-80574b497eb2 to worker[229] qsize 0
2021-05-16 11:17:19,277 DEBUG awx.main.dispatch task 28b0a3e8-b532-430d-8f48-80574b497eb2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:17:19,303 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:17:19,306 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:17:19,309 DEBUG awx.main.dispatch task 28b0a3e8-b532-430d-8f48-80574b497eb2 is finished
2021-05-16 11:17:25,137 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(25d96287-14db-4b0b-b4a3-ed3b6d262e79, queue=dev-awx-0)
[2021-05-16 11:17:25,137: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(25d96287-14db-4b0b-b4a3-ed3b6d262e79, queue=dev-awx-0)
2021-05-16 11:17:25,182 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(76caa962-0ccd-4235-8403-ade3667ed330, queue=dev-awx-0)
[2021-05-16 11:17:25,182: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(76caa962-0ccd-4235-8403-ade3667ed330, queue=dev-awx-0)
2021-05-16 11:17:25,185 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-16 11:17:25,223 DEBUG awx.main.dispatch delivered 76caa962-0ccd-4235-8403-ade3667ed330 to worker[231] qsize 0
2021-05-16 11:17:25,229 DEBUG awx.main.dispatch task 76caa962-0ccd-4235-8403-ade3667ed330 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:17:25,247 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140193118619200_5" does not exist
2021-05-16 11:17:25,254 DEBUG awx.main.dispatch task 76caa962-0ccd-4235-8403-ade3667ed330 is finished
2021-05-16 11:17:25,346 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(22670337-9b00-4d04-998c-3d103fd44f0c, queue=awx_private_queue)
[2021-05-16 11:17:25,346: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(22670337-9b00-4d04-998c-3d103fd44f0c, queue=awx_private_queue)
2021-05-16 11:17:25,383 DEBUG awx.main.dispatch delivered 22670337-9b00-4d04-998c-3d103fd44f0c to worker[229] qsize 0
2021-05-16 11:17:25,388 DEBUG awx.main.dispatch task 22670337-9b00-4d04-998c-3d103fd44f0c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:17:25,400 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:17:25,402 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:17:25,405 DEBUG awx.main.dispatch task 22670337-9b00-4d04-998c-3d103fd44f0c is finished
2021-05-16 11:17:45,360 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c1e6034d-7553-4bad-9135-4bcac0641dd0, queue=awx_private_queue)
[2021-05-16 11:17:45,360: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c1e6034d-7553-4bad-9135-4bcac0641dd0, queue=awx_private_queue)
2021-05-16 11:17:45,394 DEBUG awx.main.dispatch delivered c1e6034d-7553-4bad-9135-4bcac0641dd0 to worker[233] qsize 0
2021-05-16 11:17:45,405 DEBUG awx.main.dispatch task c1e6034d-7553-4bad-9135-4bcac0641dd0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:17:45,419 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:17:45,421 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:17:45,429 DEBUG awx.main.dispatch task c1e6034d-7553-4bad-9135-4bcac0641dd0 is finished
2021-05-16 11:17:49,248 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a6d5a0c8-e2b5-45c1-a352-be61c9640bd3, queue=awx_private_queue)
[2021-05-16 11:17:49,248: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a6d5a0c8-e2b5-45c1-a352-be61c9640bd3, queue=awx_private_queue)
2021-05-16 11:17:49,275 DEBUG awx.main.dispatch delivered a6d5a0c8-e2b5-45c1-a352-be61c9640bd3 to worker[233] qsize 0
2021-05-16 11:17:49,281 DEBUG awx.main.dispatch task a6d5a0c8-e2b5-45c1-a352-be61c9640bd3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:17:49,293 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:17:49,295 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:17:49,300 DEBUG awx.main.dispatch task a6d5a0c8-e2b5-45c1-a352-be61c9640bd3 is finished
RESULT 2
OKREADY
2021-05-16 11:18:05,357 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8557ddda-32a4-4e31-a04c-dec2aed355c6, queue=awx_private_queue)
[2021-05-16 11:18:05,357: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8557ddda-32a4-4e31-a04c-dec2aed355c6, queue=awx_private_queue)
2021-05-16 11:18:05,383 DEBUG awx.main.dispatch delivered 8557ddda-32a4-4e31-a04c-dec2aed355c6 to worker[220] qsize 0
2021-05-16 11:18:05,398 DEBUG awx.main.dispatch task 8557ddda-32a4-4e31-a04c-dec2aed355c6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:18:05,427 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:18:05,429 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:18:05,433 DEBUG awx.main.dispatch task 8557ddda-32a4-4e31-a04c-dec2aed355c6 is finished
2021-05-16 11:18:19,254 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e3c34690-31c4-4ee1-98aa-286bf156722b, queue=awx_private_queue)
[2021-05-16 11:18:19,254: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e3c34690-31c4-4ee1-98aa-286bf156722b, queue=awx_private_queue)
2021-05-16 11:18:19,282 DEBUG awx.main.dispatch delivered e3c34690-31c4-4ee1-98aa-286bf156722b to worker[220] qsize 0
2021-05-16 11:18:19,297 DEBUG awx.main.dispatch task e3c34690-31c4-4ee1-98aa-286bf156722b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:18:19,315 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:18:19,318 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:18:19,327 DEBUG awx.main.dispatch task e3c34690-31c4-4ee1-98aa-286bf156722b is finished
2021-05-16 11:18:25,136 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(76baa48f-00aa-4974-ba00-0083b81c0765, queue=dev-awx-0)
[2021-05-16 11:18:25,136: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(76baa48f-00aa-4974-ba00-0083b81c0765, queue=dev-awx-0)
2021-05-16 11:18:25,171 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-16 11:18:25,182 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c7c1cd3d-35d8-47b4-a5d4-499e834d1b9d, queue=dev-awx-0)
[2021-05-16 11:18:25,182: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c7c1cd3d-35d8-47b4-a5d4-499e834d1b9d, queue=dev-awx-0)
2021-05-16 11:18:25,201 DEBUG awx.main.dispatch delivered c7c1cd3d-35d8-47b4-a5d4-499e834d1b9d to worker[233] qsize 0
2021-05-16 11:18:25,208 DEBUG awx.main.dispatch task c7c1cd3d-35d8-47b4-a5d4-499e834d1b9d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:18:25,229 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140193118619200_5" does not exist
2021-05-16 11:18:25,235 DEBUG awx.main.dispatch task c7c1cd3d-35d8-47b4-a5d4-499e834d1b9d is finished
2021-05-16 11:18:25,357 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(65ad023f-4caf-45b2-8507-abc834753efa, queue=awx_private_queue)
[2021-05-16 11:18:25,357: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(65ad023f-4caf-45b2-8507-abc834753efa, queue=awx_private_queue)
2021-05-16 11:18:25,382 DEBUG awx.main.dispatch delivered 65ad023f-4caf-45b2-8507-abc834753efa to worker[231] qsize 0
2021-05-16 11:18:25,389 DEBUG awx.main.dispatch task 65ad023f-4caf-45b2-8507-abc834753efa starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:18:25,408 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:18:25,409 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:18:25,413 DEBUG awx.main.dispatch task 65ad023f-4caf-45b2-8507-abc834753efa is finished
2021-05-16 11:18:45,365 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bd155e93-1ad0-485b-8cc4-8891a3fcf1d5, queue=awx_private_queue)
[2021-05-16 11:18:45,365: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bd155e93-1ad0-485b-8cc4-8891a3fcf1d5, queue=awx_private_queue)
2021-05-16 11:18:45,396 DEBUG awx.main.dispatch delivered bd155e93-1ad0-485b-8cc4-8891a3fcf1d5 to worker[233] qsize 0
2021-05-16 11:18:45,405 DEBUG awx.main.dispatch task bd155e93-1ad0-485b-8cc4-8891a3fcf1d5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:18:45,416 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:18:45,417 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:18:45,422 DEBUG awx.main.dispatch task bd155e93-1ad0-485b-8cc4-8891a3fcf1d5 is finished
2021-05-16 11:18:49,252 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8bd5343f-f4ef-4b24-8ed3-857962dbc5ef, queue=awx_private_queue)
[2021-05-16 11:18:49,252: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8bd5343f-f4ef-4b24-8ed3-857962dbc5ef, queue=awx_private_queue)
2021-05-16 11:18:49,280 DEBUG awx.main.dispatch delivered 8bd5343f-f4ef-4b24-8ed3-857962dbc5ef to worker[220] qsize 0
2021-05-16 11:18:49,288 DEBUG awx.main.dispatch task 8bd5343f-f4ef-4b24-8ed3-857962dbc5ef starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:18:49,301 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:18:49,303 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:18:49,310 DEBUG awx.main.dispatch task 8bd5343f-f4ef-4b24-8ed3-857962dbc5ef is finished
RESULT 2
OKREADY
2021-05-16 11:19:05,367 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(606113f2-0a61-41a6-a90b-2e4214d36ab9, queue=awx_private_queue)
[2021-05-16 11:19:05,367: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(606113f2-0a61-41a6-a90b-2e4214d36ab9, queue=awx_private_queue)
2021-05-16 11:19:05,394 DEBUG awx.main.dispatch delivered 606113f2-0a61-41a6-a90b-2e4214d36ab9 to worker[231] qsize 0
2021-05-16 11:19:05,402 DEBUG awx.main.dispatch task 606113f2-0a61-41a6-a90b-2e4214d36ab9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:19:05,431 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:19:05,434 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:19:05,438 DEBUG awx.main.dispatch task 606113f2-0a61-41a6-a90b-2e4214d36ab9 is finished
2021-05-16 11:19:19,259 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dc000269-2bb9-4ddc-8b18-3c6fc41c8837, queue=awx_private_queue)
[2021-05-16 11:19:19,259: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dc000269-2bb9-4ddc-8b18-3c6fc41c8837, queue=awx_private_queue)
2021-05-16 11:19:19,295 DEBUG awx.main.dispatch delivered dc000269-2bb9-4ddc-8b18-3c6fc41c8837 to worker[229] qsize 0
2021-05-16 11:19:19,306 DEBUG awx.main.dispatch task dc000269-2bb9-4ddc-8b18-3c6fc41c8837 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:19:19,333 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:19:19,337 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-16 11:19:19,345 DEBUG awx.main.dispatch task dc000269-2bb9-4ddc-8b18-3c6fc41c8837 is finished
2021-05-16 11:19:25,137 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(4f10c93b-836c-411b-aa01-62dac55f7fca, queue=dev-awx-0)
[2021-05-16 11:19:25,137: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(4f10c93b-836c-411b-aa01-62dac55f7fca, queue=dev-awx-0)
2021-05-16 11:19:25,170 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-16 11:19:25,182 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(6d09226c-30a4-412d-89c9-7cd70b24cc03, queue=dev-awx-0)
[2021-05-16 11:19:25,182: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(6d09226c-30a4-412d-89c9-7cd70b24cc03, queue=dev-awx-0)
2021-05-16 11:19:25,204 DEBUG awx.main.dispatch delivered 6d09226c-30a4-412d-89c9-7cd70b24cc03 to worker[220] qsize 0
2021-05-16 11:19:25,212 DEBUG awx.main.dispatch task 6d09226c-30a4-412d-89c9-7cd70b24cc03 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-16 11:19:25,234 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_140193118619200_4" does not exist
2021-05-16 11:19:25,238 DEBUG awx.main.dispatch task 6d09226c-30a4-412d-89c9-7cd70b24cc03 is finished
2021-05-16 11:19:25,367 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2f180a48-f550-4578-82cb-7bf5259edfee, queue=awx_private_queue)
[2021-05-16 11:19:25,367: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2f180a48-f550-4578-82cb-7bf5259edfee, queue=awx_private_queue)
2021-05-16 11:19:25,402 DEBUG awx.main.dispatch delivered 2f180a48-f550-4578-82cb-7bf5259edfee to worker[229] qsize 0
2021-05-16 11:19:25,410 DEBUG awx.main.dispatch task 2f180a48-f550-4578-82cb-7bf5259edfee starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:19:25,437 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:19:25,439 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:19:25,444 DEBUG awx.main.dispatch task 2f180a48-f550-4578-82cb-7bf5259edfee is finished
2021-05-16 11:19:45,377 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b4668ad4-5a04-44e5-9318-4a1c995ddf30, queue=awx_private_queue)
[2021-05-16 11:19:45,377: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b4668ad4-5a04-44e5-9318-4a1c995ddf30, queue=awx_private_queue)
2021-05-16 11:19:45,401 DEBUG awx.main.dispatch delivered b4668ad4-5a04-44e5-9318-4a1c995ddf30 to worker[233] qsize 0
2021-05-16 11:19:45,413 DEBUG awx.main.dispatch task b4668ad4-5a04-44e5-9318-4a1c995ddf30 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-16 11:19:45,436 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-16 11:19:45,438 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-16 11:19:45,443 DEBUG awx.main.dispatch task b4668ad4-5a04-44e5-9318-4a1c995ddf30 is finished
2021-05-16 11:19:49,254 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(95ad9b54-5d0f-4342-b6f2-556470b28b0b, queue=awx_private_queue)
[2021-05-16 11:19:49,254: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(95ad9b54-5d0f-4342-b6f2-556470b28b0b, queue=awx_private_queue)
2021-05-16 11:19:49,279 DEBUG awx.main.dispatch delivered 95ad9b54-5d0f-4342-b6f2-556470b28b0b to worker[220] qsize 0
2021-05-16 11:19:49,289 DEBUG awx.main.dispatch task 95ad9b54-5d0f-4342-b6f2-556470b28b0b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-16 11:19:49,305 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-16 11:19:49,311 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.